Skip to content

Commit

Permalink
Modify sample to use uriBaseIds (#2002)
Browse files Browse the repository at this point in the history 
* Modify sample to use uriBaseIds.

* Add TryReconstructAbsoluteUri unit test for missing trailing slash.

* Shorten and move comment.

* Introduce string constants.

* Add SarifLogger tests for run enhancement.

* Add a period.

* Test population of artifact contents in presence of uriBaseId.

* Add tests for GetEncodingFromName.

* Remove renamed-and-mostly-changed file.

* Fix file-scheme-related bug in UriConverter.

* Test for analysis targets with encoding and contents.

* DRY out "file" scheme constant.

* Mentioned fix for #2001 in release history.

* Remove extra blank line.

* Fix typo in comment.

* Fix another typo.

* Add version control provenance; change to REPO_ROOT.

* Visit results to provide region snippets.

* Clean up InsertOptionalDataVisitorTests

* Add unit test for visiting individual result.

* Add rule help URIs to test data.

Co-authored-by: Larry Golding <lgolding@comcast.net>
Co-authored-by: Michael C. Fanning <michael.fanning@microsoft.com>
  • Loading branch information
@michaelcfanning
3 people authored Jul 20, 2020
1 parent f4cf16e commit b857f62
Show file tree
Hide file tree
Showing 20 changed files with 644 additions and 142 deletions.
4 changes: 3 additions & 1 deletion src/ReleaseHistory.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# SARIF Package Release History (SDK, Driver, Converters, and Multitool)

## **v2.3.3** [Sdk](https://www.nuget.org/packages/Sarif.Sdk/2.3.3) | [Driver](https://www.nuget.org/packages/Sarif.Driver/2.3.3) | [Converters](https://www.nuget.org/packages/Sarif.Converters/2.3.3) | [Multitool](https://www.nuget.org/packages/Sarif.Multitool/2.3.3)
* FEATURE: Improve `SarifSdkSample` application: use `uriBaseIds`.
* BUGFIX: If you created a URI from an absolute file path (for example, `C:\test\file.c`), then it would be serialized with that exact string, which is not a valid URI. This is now fixed. [#2001](https://github.com/microsoft/sarif-sdk/issues/2001)
* FEATURE: Add additional checks to SARIF analysis rule `SARIF2004.OptimizeFileSize`.

## **v2.3.2** [Sdk](https://www.nuget.org/packages/Sarif.Sdk/2.3.2) | [Driver](https://www.nuget.org/packages/Sarif.Driver/2.3.2) | [Converters](https://www.nuget.org/packages/Sarif.Converters/2.3.2) | [Multitool](https://www.nuget.org/packages/Sarif.Multitool/2.3.2)
Expand All @@ -19,7 +21,7 @@
* BUGFIX: In validation rules, `shortDescription` is now calculated by `GetFirstSentence` method, fixing a bug in sentence breaking. [#1887](https://github.com/microsoft/sarif-sdk/issues/1887)
* BUGFIX: `WorkItemFiler` now logs correctly the details for `LogMetricsForProcessedModel` method [#1896](https://github.com/microsoft/sarif-sdk/issues/1896)
* FEATURE: Add validation rule `SARIF1019`, which requires every result to have at least one of `result.ruleId` and `result.rule.id`. If both are present, they must be equal. [#1880](https://github.com/microsoft/sarif-sdk/issues/1880)
* FEATURE: Add validation rule `SARIF1020`, which requires that the $schema property should be present, and must refer to the final version of the SARIF 2.1.0 schema. [#1890](https://github.com/microsoft/sarif-sdk/issues/1890)
* FEATURE: Add validation rule `SARIF1020`, which requires that the $schema property should be present, and must refer to the final version of the SARIF 2.1.0 schema. [#1890](https://github.com/microsoft/sarif-sdk/issues/1890)
* FEATURE: Expose `Run.MergeResultsFrom(Run)` to merge Results from multiple Runs using code from result matching algorithm.
* BREAKING: Rename `RemapIndicesVisitor` to `RunMergingVisitor` and redesign to control how much merging occurs internally.

Expand Down
82 changes: 66 additions & 16 deletions src/Samples/Sarif.Sdk.Sample/Program.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,17 @@
using System.Text;
using CommandLine;
using Microsoft.CodeAnalysis.Sarif;
using Microsoft.CodeAnalysis.Sarif.Readers;
using Microsoft.CodeAnalysis.Sarif.Writers;
using Newtonsoft.Json;

namespace Sarif.Sdk.Sample
{
public class Program
{
static int Main(string[] args)
private const string RepoRootBaseId = "REPO_ROOT";
private const string BinRootBaseId = "BIN_ROOT";

internal static int Main(string[] args)
{
int result = Parser.Default.ParseArguments<LoadOptions, CreateOptions>(args)
.MapResult(
Expand All @@ -31,7 +33,7 @@ static int Main(string[] args)
/// </summary>
/// <param name="options">Load verb options.</param>
/// <returns>Exit code</returns>
static int LoadSarifLogFile(LoadOptions options)
internal static int LoadSarifLogFile(LoadOptions options)
{
string logText = File.ReadAllText(options.InputFilePath);
SarifLog log = JsonConvert.DeserializeObject<SarifLog>(logText);
Expand All @@ -46,12 +48,18 @@ static int LoadSarifLogFile(LoadOptions options)
/// </summary>
/// <param name="options">Create verb options.</param>
/// <returns>Exit code</returns>
static int CreateSarifLogFile(CreateOptions options)
internal static int CreateSarifLogFile(CreateOptions options)
{
// We'll use this source file for several defect results -- the
// SampleSourceFiles folder should be a child of the project folder,
// two levels up from the folder that contains the EXE (e.g., bin\Debug).
var artifactLocation = new ArtifactLocation { Uri = new Uri($"file://{AppDomain.CurrentDomain.BaseDirectory}../../SampleSourceFiles/AnalysisSample.cs") };
string scanRootDirectory = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"..\..\SampleSourceFiles\"));
var scanRootUri = new Uri(scanRootDirectory, UriKind.Absolute);
var artifactLocation = new ArtifactLocation
{
Uri = new Uri("AnalysisSample.cs", UriKind.Relative),
UriBaseId = RepoRootBaseId
};

// Create a list of rules that will be enforced during your analysis
#region Rules list
Expand All @@ -71,7 +79,8 @@ static int CreateSarifLogFile(CreateOptions options)
Text = "The property {0} returns an array."
}
}
}
},
HelpUri = new Uri("https://www.example.com/rules/CA1819")
},
new ReportingDescriptor
{
Expand All @@ -87,7 +96,8 @@ static int CreateSarifLogFile(CreateOptions options)
Text = "The test for an empty string is performed by a string comparison rather than by testing String.Length."
}
}
}
},
HelpUri = new Uri("https://www.example.com/rules/CA1820")
},
new ReportingDescriptor
{
Expand All @@ -103,7 +113,8 @@ static int CreateSarifLogFile(CreateOptions options)
Text = "The array-valued field {0} is marked readonly."
}
}
}
},
HelpUri = new Uri("https://www.example.com/rules/CA2105")
},
new ReportingDescriptor
{
Expand All @@ -119,7 +130,8 @@ static int CreateSarifLogFile(CreateOptions options)
Text = "The Dispose method does not call the base class Dispose method."
}
}
}
},
HelpUri = new Uri("https://www.example.com/rules/CA2215")
}
};
#endregion
Expand Down Expand Up @@ -247,6 +259,38 @@ static int CreateSarifLogFile(CreateOptions options)
};
#endregion

string binRootDirectory = @"d:\src\module\";
var binRootUri = new Uri(binRootDirectory, UriKind.Absolute);

var run = new Run
{
OriginalUriBaseIds = new Dictionary<string, ArtifactLocation>
{
[RepoRootBaseId] = new ArtifactLocation
{
Uri = scanRootUri
},
[BinRootBaseId] = new ArtifactLocation
{
Uri = binRootUri
}
},
VersionControlProvenance = new VersionControlDetails[]
{
new VersionControlDetails
{
RepositoryUri = new Uri("https://github.com/microsoft/sarif-sdk"),
RevisionId = "ee5a1ca8",
Branch = "master",
MappedTo = new ArtifactLocation
{
Uri = new Uri(".", UriKind.Relative),
UriBaseId = RepoRootBaseId
}
}
}
};

// The SarifLogger will write the JSON-formatted log to this StringBuilder
var sb = new StringBuilder();

Expand All @@ -257,9 +301,10 @@ static int CreateSarifLogFile(CreateOptions options)
loggingOptions: LoggingOptions.PrettyPrint, // Use PrettyPrint to generate readable (multi-line, indented) JSON
dataToInsert:
OptionallyEmittedData.TextFiles | // Embed source file content directly in the log file -- great for portability of the log!
OptionallyEmittedData.Hashes,
OptionallyEmittedData.Hashes |
OptionallyEmittedData.RegionSnippets,
tool: null,
run: null,
run: run,
analysisTargets: null,
invocationTokensToRedact: null,
invocationPropertiesToLog: null,
Expand All @@ -274,7 +319,11 @@ static int CreateSarifLogFile(CreateOptions options)
var result = new Result()
{
RuleId = rule.Id,
AnalysisTarget = new ArtifactLocation { Uri = new Uri(@"file://d:/src/module/example.dll") }, // This is the file that was analyzed
AnalysisTarget = new ArtifactLocation
{
Uri = new Uri("example.dll", UriKind.Relative), // This is the file that was analyzed
UriBaseId = BinRootBaseId
},
Message = new Message
{
Id = "Default",
Expand Down Expand Up @@ -302,7 +351,8 @@ static int CreateSarifLogFile(CreateOptions options)
{
// Because this file doesn't exist, it will be included in the files list but will only have a path and MIME type
// This is the behavior you'll see any time a file can't be located/accessed
Uri = new Uri($"file://{AppDomain.CurrentDomain.BaseDirectory}/../../../SampleSourceFiles/SomeOtherSourceFile.cs"),
Uri = new Uri("SomeOtherSourceFile.cs", UriKind.Relative),
UriBaseId = RepoRootBaseId
},
Region = new Region
{
Expand Down Expand Up @@ -330,7 +380,7 @@ static int CreateSarifLogFile(CreateOptions options)
ArtifactLocation = artifactLocation,
Region = new Region
{
StartLine = 212
StartLine = 17
}
}
}
Expand All @@ -345,7 +395,7 @@ static int CreateSarifLogFile(CreateOptions options)
ArtifactLocation = artifactLocation,
Region = new Region
{
StartLine = 452 // Fake example
StartLine = 24 // Fake example
}
}
}
Expand All @@ -360,7 +410,7 @@ static int CreateSarifLogFile(CreateOptions options)
ArtifactLocation = artifactLocation,
Region = new Region
{
StartLine = 145
StartLine = 26 // Fake example
}
}
}
Expand Down
9 changes: 4 additions & 5 deletions src/Samples/Sarif.Sdk.Sample/Properties/AssemblyInfo.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,17 @@
// license. See LICENSE file in the project root for full license information.

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("SARIF SDK v2.0 Sample")]
[assembly: AssemblyDescription("SARIF v2.0 log file read/write sample application")]
[assembly: AssemblyTitle("SARIF SDK v2.1.0 Sample")]
[assembly: AssemblyDescription("SARIF v2.1.0 log file read/write sample application")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("SARIF SDK v2.0")]
[assembly: AssemblyCopyright("Copyright © 2018")]
[assembly: AssemblyProduct("SARIF SDK v2.1.0")]
[assembly: AssemblyCopyright("Copyright © 2018")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]

Expand Down
4 changes: 2 additions & 2 deletions src/Sarif.WorkItems/SarifWorkItemFiler.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ public virtual SarifLog FileWorkItems(Uri sarifLogFileLocation)
{
sarifLogFileLocation = sarifLogFileLocation ?? throw new ArgumentNullException(nameof(sarifLogFileLocation));

if (sarifLogFileLocation.IsAbsoluteUri && sarifLogFileLocation.Scheme == "file")
if (sarifLogFileLocation.IsAbsoluteUri && sarifLogFileLocation.Scheme == UriUtilities.FileScheme)
{
if (sarifLogFileLocation.IsAbsoluteUri && sarifLogFileLocation.Scheme == "file:")
if (sarifLogFileLocation.IsAbsoluteUri && sarifLogFileLocation.Scheme == UriUtilities.FileScheme.WithColon())
{
using (var stream = new FileStream(sarifLogFileLocation.LocalPath, FileMode.Open, FileAccess.Read))
using (var reader = new StreamReader(stream))
Expand Down
4 changes: 3 additions & 1 deletion src/Sarif/Core/ArtifactLocation.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ public bool TryReconstructAbsoluteUri(IDictionary<string, ArtifactLocation> orig
// I'd like to use the ctor new Uri(baseUri, relativeUri) here, but it fails with
// ArgumentOutOfRangeException, perhaps because it doesn't like the baseUri argument
// to be relative. So...
stemUri = new Uri(artifactLocation.Uri.OriginalString + stemUri.OriginalString, UriKind.RelativeOrAbsolute);
string artifactLocationOriginalUriString = artifactLocation.Uri.OriginalString;
if (!artifactLocationOriginalUriString.EndsWith("/")) { artifactLocationOriginalUriString += "/"; }
stemUri = new Uri(artifactLocationOriginalUriString + stemUri.OriginalString, UriKind.RelativeOrAbsolute);
}

// If we got here, we found an absolute URI.
Expand Down
52 changes: 28 additions & 24 deletions src/Sarif/Core/Run.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ public partial class Run
private static readonly Invocation EmptyInvocation = new Invocation();
private static readonly LogicalLocation EmptyLogicalLocation = new LogicalLocation();

private IDictionary<ArtifactLocation, int> _fileToIndexMap;
private IDictionary<ArtifactLocation, int> _artifactLocationToIndexMap;

public Uri ExpandUrisWithUriBaseId(string key, string currentValue = null)
{
Expand Down Expand Up @@ -47,7 +47,7 @@ public int GetFileIndex(
}
}

if (_fileToIndexMap == null)
if (_artifactLocationToIndexMap == null)
{
InitializeFileToIndexMap();
}
Expand All @@ -58,11 +58,11 @@ public int GetFileIndex(
return fileLocation.Index;
}

// Strictly speaking, some elements that may contribute to a files table
// key are case sensitive, e.g., everything but the schema and protocol of a
// web URI. We don't have a proper comparer implementation that can handle
// Strictly speaking, some elements that may contribute to a files table
// key are case sensitive, e.g., everything but the scheme and protocol of a
// web URI. We don't have a proper comparer implementation that can handle
// all cases. For now, we cover the Windows happy path, which assumes that
// most URIs in log files are file paths (which are case-insensitive)
// most URIs in log files are file paths (which are case-insensitive).
//
// Tracking item for an improved comparer:
// https://github.com/Microsoft/sarif-sdk/issues/973
Expand All @@ -76,61 +76,65 @@ public int GetFileIndex(
// throughout the emitted log.
fileLocation.Uri = new Uri(UriHelper.MakeValidUri(fileLocation.Uri.OriginalString), UriKind.RelativeOrAbsolute);

var filesTableKey = new ArtifactLocation
var artifactLocation = new ArtifactLocation
{
Uri = fileLocation.Uri,
UriBaseId = fileLocation.UriBaseId
};

if (!_fileToIndexMap.TryGetValue(filesTableKey, out int fileIndex))
if (!_artifactLocationToIndexMap.TryGetValue(artifactLocation, out int artifactIndex))
{
if (addToFilesTableIfNotPresent)
{
this.Artifacts = this.Artifacts ?? new List<Artifact>();
fileIndex = this.Artifacts.Count;
artifactIndex = this.Artifacts.Count;

var fileData = Artifact.Create(
filesTableKey.Uri,
Uri artifactUri = artifactLocation.TryReconstructAbsoluteUri(this.OriginalUriBaseIds, out Uri resolvedUri)
? resolvedUri
: artifactLocation.Uri;

var artifact = Artifact.Create(
artifactUri,
dataToInsert,
hashData: hashData,
encoding: null);
encoding: encoding);

// Copy ArtifactLocation to ensure changes to Result copy don't affect new Run.Artifacts copy
fileData.Location = new ArtifactLocation(fileLocation);
artifact.Location = new ArtifactLocation(fileLocation);

this.Artifacts.Add(fileData);
this.Artifacts.Add(artifact);

_fileToIndexMap[filesTableKey] = fileIndex;
_artifactLocationToIndexMap[artifactLocation] = artifactIndex;
}
else
{
// We did not find the item. The call was not configured to add the entry.
// Return the default value that indicates the item isn't present.
fileIndex = -1;
artifactIndex = -1;
}
}

fileLocation.Index = fileIndex;
return fileIndex;
fileLocation.Index = artifactIndex;
return artifactIndex;
}

private void InitializeFileToIndexMap()
{
_fileToIndexMap = new Dictionary<ArtifactLocation, int>(ArtifactLocation.ValueComparer);
_artifactLocationToIndexMap = new Dictionary<ArtifactLocation, int>(ArtifactLocation.ValueComparer);

// First, we'll initialize our file object to index map
// with any files that already exist in the table
for (int i = 0; i < this.Artifacts?.Count; i++)
{
Artifact fileData = this.Artifacts[i];
Artifact artifact = this.Artifacts[i];

var fileLocation = new ArtifactLocation
var artifactLocation = new ArtifactLocation
{
Uri = fileData.Location?.Uri,
UriBaseId = fileData.Location?.UriBaseId,
Uri = artifact.Location?.Uri,
UriBaseId = artifact.Location?.UriBaseId,
};

_fileToIndexMap[fileLocation] = i;
_artifactLocationToIndexMap[artifactLocation] = i;
}
}

Expand Down
Loading

0 comments on commit b857f62

Please sign in to comment.