Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enqueue files filter #2599

Merged
merged 16 commits into from
Dec 29, 2022
Merged

Enqueue files filter #2599

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
26a4291
Add overridable filtering mechanism to driver file enumeration phase.
michaelcfanning Dec 28, 2022
c67b6dd
Update max file size type. Obsolete AnalyzeCommandBase.
michaelcfanning Dec 28, 2022
7703770
Resolve hangs in file enumeration phase.
michaelcfanning Dec 28, 2022
c1e8330
Merge from main
michaelcfanning Dec 28, 2022
9baabcb
Test fixes.
michaelcfanning Dec 28, 2022
24dedbd
Restore more tests to working order.
michaelcfanning Dec 28, 2022
e50fb2e
All tests now passing.
michaelcfanning Dec 28, 2022
47a0c26
Updates after first code review.
michaelcfanning Dec 28, 2022
de74667
Add new analyze context base file.
michaelcfanning Dec 28, 2022
8877214
Make max file size arg overridable.
michaelcfanning Dec 28, 2022
ee6c142
Merge branch 'main' into enqueue-files-filter
michaelcfanning Dec 28, 2022
a68651f
Consult context for maf file size not options.
michaelcfanning Dec 28, 2022
7af863f
Merge branch 'enqueue-files-filter' of https://github.com/microsoft/s…
michaelcfanning Dec 28, 2022
0c60205
Ignored files count should be restricted to those that exceed file si…
michaelcfanning Dec 28, 2022
50642ab
File size test fixups due to default arg behavior change.
michaelcfanning Dec 29, 2022
6e658bb
File size test fixups due to default arg behavior change.
michaelcfanning Dec 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 15 additions & 11 deletions src/ReleaseHistory.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,22 @@

## **v3.2.0** (UNRELEASED)

* BREAKING: Correct casing of `LogMissingreportingConfiguration` helper to `LogMissingReportingConfiguration`.
* BREAKING: Change type of `MaxFileSizeInKilobytes` from int to long in `IAnalysisContext` and other classes.
* BREAKING: For `Guid` properties defined in SARIF spec, updated Json schema to use `uuid`, and updated C# object model to use `Guid?` instead of `string`. [#2555](https://github.com/microsoft/sarif-sdk/pull/2555)
* FEATURE: Provide mechanism to populate `SarifLogger` with a `FileRegionsCache` instance.
* BUGFIX: Another attempt to resolve 'InvalidOperationException' with message `Collection was modified; enumeration operation may not execute` in `MultithreadedAnalyzeCommandBase`, raised when analyzing with the `--hashes` switch. [#2459](https://github.com/microsoft/sarif-sdk/pull/2549). There was a previous attempt to fix this in [#2447](https://github.com/microsoft/sarif-sdk/pull/2447).
* FEATURE: Allow initialization of file regions cache in `InsertOptionalDataVisitor` (previously initialized exclusively from `FileRegionsCache.Instance`).
* BUGFIX: Resolve issue where `match-results-forward` command fails to generate VersionControlDetails data. [#2487](https://github.com/microsoft/sarif-sdk/pull/2487)
* BUGFIX: Remove duplicated rule definitions when executing `match-results-forward` commands for results with sub-rule ids. [#2486](https://github.com/microsoft/sarif-sdk/pull/2486)
* BUGFIX: Update `merge` command to properly produce runs by tool and version when passed the `--merge-runs` argument. [#2488](https://github.com/microsoft/sarif-sdk/pull/2488)
* BUGFIX: Eliminate `IOException` and `DirectoryNotFoundException` exceptions thrown by `merge` command when splitting by rule (due to invalid file characters in rule ids). [#2513](https://github.com/microsoft/sarif-sdk/pull/2513)
* BUGFIX: Fix classes inside NotYetAutoGenerated folder missing `virtual` keyword for public methods and properties, by regenerate and manually sync the changes. [#2537](https://github.com/microsoft/sarif-sdk/pull/2537)
* FEATURE: Allow external set of `MaxFileSizeInKilobytes`, which will allow SDK users to change the value. (Default value is 1024) [#2578](https://github.com/microsoft/sarif-sdk/pull/2578)
* FEATURE: Add a Github validation rule `GH1007`, which requires flattened result message so GHAS code scanning can ingest the log. [#2580](https://github.com/microsoft/sarif-sdk/issues/2580)
* BUGFIX: MSBuild Converter now accepts case insensitive keywords and supports PackageValidator msbuild log output. [#2579](https://github.com/microsoft/sarif-sdk/pull/2579)
* BUGFIX : Resolve hangs due to unhandled exceptions during multithreaded analysis file enumeration phase.
* BUGFIX : Another attempt to resolve 'InvalidOperationException' with message `Collection was modified; enumeration operation may not execute` in `MultithreadedAnalyzeCommandBase`, raised when analyzing with the `--hashes` switch. [#2459](https://github.com/microsoft/sarif-sdk/pull/2549). There was a previous attempt to fix this in [#2447](https://github.com/microsoft/sarif-sdk/pull/2447).
* BUGFIX : Resolve issue where `match-results-forward` command fails to generate VersionControlDetails data. [#2487](https://github.com/microsoft/sarif-sdk/pull/2487)
* BUGFIX : Remove duplicated rule definitions when executing `match-results-forward` commands for results with sub-rule ids. [#2486](https://github.com/microsoft/sarif-sdk/pull/2486)
* BUGFIX : Update `merge` command to properly produce runs by tool and version when passed the `--merge-runs` argument. [#2488](https://github.com/microsoft/sarif-sdk/pull/2488)
* BUGFIX : Eliminate `IOException` and `DirectoryNotFoundException` exceptions thrown by `merge` command when splitting by rule (due to invalid file characters in rule ids). [#2513](https://github.com/microsoft/sarif-sdk/pull/2513)
* BUGFIX : Fix classes inside NotYetAutoGenerated folder missing `virtual` keyword for public methods and properties, by regenerate and manually sync the changes. [#2537](https://github.com/microsoft/sarif-sdk/pull/2537)
* BUGFIX : MSBuild Converter now accepts case insensitive keywords and supports PackageValidator msbuild log output. [#2579](https://github.com/microsoft/sarif-sdk/pull/2579)
* FEATURE : Provide mechanism to populate `SarifLogger` with a `FileRegionsCache` instance.
* FEATURE : Provide overridable `ShouldEnqueue` predicate method to filter files from driver processing.
* FEATURE : Allow external set of `MaxFileSizeInKilobytes`, which will allow SDK users to change the value. (Default value is 1024) [#2578](https://github.com/microsoft/sarif-sdk/pull/2578)
* FEATURE : Add a Github validation rule `GH1007`, which requires flattened result message so GHAS code scanning can ingest the log. [#2580](https://github.com/microsoft/sarif-sdk/issues/2580)
* FEATURE : Allow initialization of file regions cache in `InsertOptionalDataVisitor` (previously initialized exclusively from `FileRegionsCache.Instance`).

## **v3.1.0** [Sdk](https://www.nuget.org/packages/Sarif.Sdk/3.1.0) | [Driver](https://www.nuget.org/packages/Sarif.Driver/3.1.0) | [Converters](https://www.nuget.org/packages/Sarif.Converters/3.1.0) | [Multitool](https://www.nuget.org/packages/Sarif.Multitool/3.1.0) | [Multitool Library](https://www.nuget.org/packages/Sarif.Multitool.Library/3.1.0)

Expand Down
25 changes: 21 additions & 4 deletions src/Sarif.Driver/Sdk/AnalyzeCommandBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

namespace Microsoft.CodeAnalysis.Sarif.Driver
{
[Obsolete("AnalyzeCommandBase will be deprecated entirely soon. Use MultithreadedAnalyzeCommandBase instead.")]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AnalyzeCommandBase

It grows wearisome to maintain this obsolete single-threaded analysis, let's zap it next release.

public abstract class AnalyzeCommandBase<TContext, TOptions> : PluginDriverCommand<TOptions>
where TContext : IAnalysisContext, new()
where TOptions : AnalyzeOptionsBase
Expand Down Expand Up @@ -97,12 +98,13 @@ public override int Run(TOptions options)
catch (Exception ex)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogUnhandledEngineException

This helper previously returned an error condition to OR into RuntimeErrors. I've updated the helper to be consistent with other utility methods. We now retrieve the RuntimeErrors in the finally block.

{
// These exceptions escaped our net and must be logged here
RuntimeErrors |= Errors.LogUnhandledEngineException(_rootContext, ex);
Errors.LogUnhandledEngineException(_rootContext, ex);
ExecutionException = ex;
return FAILURE;
}
finally
{
RuntimeErrors |= _rootContext.RuntimeErrors;
logger.AnalysisStopped(RuntimeErrors);
}
}
Expand Down Expand Up @@ -217,6 +219,18 @@ protected virtual void ValidateOptions(TContext context, TOptions analyzeOptions
}
}

protected virtual bool ShouldEnqueue(string file, TContext context)
{
bool shouldEnqueue = IsTargetWithinFileSizeLimit(file, context.MaxFileSizeInKilobytes, out long fileSizeInKb);

if (!shouldEnqueue)
{
Warnings.LogFileSkippedDueToSize(context, file, fileSizeInKb);
}

return shouldEnqueue;
}

internal AggregatingLogger InitializeLogger(AnalyzeOptionsBase analyzeOptions)
{
_tool = Tool.CreateFromAssemblyData();
Expand Down Expand Up @@ -257,7 +271,7 @@ private ISet<string> CreateTargetsSet(TOptions analyzeOptions)
foreach (string file in fileSpecifier.Files)
{
// Only include files that are below the max size limit.
if (IsTargetWithinFileSizeLimit(file, _rootContext.MaxFileSizeInKilobytes))
if (ShouldEnqueue(file, _rootContext))
{
targets.Add(file);
}
Expand Down Expand Up @@ -288,10 +302,13 @@ protected virtual TContext CreateContext(
{
Logger = logger,
RuntimeErrors = runtimeErrors,
Policy = policy
Policy = policy ?? new PropertiesDictionary()
};

context.MaxFileSizeInKilobytes = options.MaxFileSizeInKilobytes;
context.MaxFileSizeInKilobytes =
options.MaxFileSizeInKilobytes >= 0
? options.MaxFileSizeInKilobytes
: 1024;

if (filePath != null)
{
Expand Down
11 changes: 9 additions & 2 deletions src/Sarif.Driver/Sdk/AnalyzeOptionsBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ namespace Microsoft.CodeAnalysis.Sarif.Driver
[Verb("analyze", HelpText = "Analyze one or more binary files for security and correctness issues.")]
public abstract class AnalyzeOptionsBase : CommonOptionsBase
{
public AnalyzeOptionsBase()
{
Kind = new List<ResultKind> { ResultKind.Fail };
Level = new List<FailureLevel> { FailureLevel.Warning, FailureLevel.Error };
MaxFileSizeInKilobytes = AnalyzerContextBase.MaxFileSizeInKilobytesDefaultValue;
}

[Value(0,
HelpText = "One or more specifiers to a file, directory, or filter pattern that resolves to one or more binaries to analyze.")]
public IEnumerable<string> TargetFileSpecifiers { get; set; }
Expand Down Expand Up @@ -118,7 +125,7 @@ public abstract class AnalyzeOptionsBase : CommonOptionsBase
[Option(
"max-file-size-in-kb",
HelpText = "The maximum file size (in kilobytes) that will be analyzed.",
Default = 1024)]
public int MaxFileSizeInKilobytes { get; set; } = 1024;
Default = AnalyzerContextBase.MaxFileSizeInKilobytesDefaultValue)]
public long MaxFileSizeInKilobytes { get; set; }
}
}
10 changes: 8 additions & 2 deletions src/Sarif.Driver/Sdk/ExportConfigurationCommandBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,17 @@ public override int Run(ExportConfigurationOptions exportOptions)

try
{
PropertiesDictionary allOptions = new PropertiesDictionary();
var allOptions = new PropertiesDictionary();

// The export command could be updated in the future to accept an arbitrary set
// of analyzers for which to build an options XML file suitable for configuring them.
ImmutableArray<IOptionsProvider> providers = CompositionUtilities.GetExports<IOptionsProvider>(RetrievePluginAssemblies(DefaultPluginAssemblies, exportOptions.PluginFilePaths));
var providers = new List<IOptionsProvider>(CompositionUtilities.GetExports<IOptionsProvider>(RetrievePluginAssemblies(DefaultPluginAssemblies, exportOptions.PluginFilePaths)));

if (AdditionalOptionsProvider != null)
{
providers.Add(AdditionalOptionsProvider);
}

foreach (IOptionsProvider provider in providers)
{
IOption sampleOption = null;
Expand Down
Loading