Enqueue files filter

src/ReleaseHistory.md

@@ -2,25 +2,34 @@
 
 ## **v3.2.0** (UNRELEASED)
 
+* BREAKING: Correct casing of `LogMissingreportingConfiguration` helper to `LogMissingReportingConfiguration`. [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* BREAKING: Change type of `MaxFileSizeInKilobytes` from int to long in `IAnalysisContext` and other classes.  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
 * BREAKING: For `Guid` properties defined in SARIF spec, updated Json schema to use `uuid`, and updated C# object model to use `Guid?` instead of `string`. [#2555](https://github.com/microsoft/sarif-sdk/pull/2555)
-* FEATURE: Provide mechanism to populate `SarifLogger` with a `FileRegionsCache` instance.
-* BUGFIX: Another attempt to resolve 'InvalidOperationException' with message `Collection was modified; enumeration operation may not execute` in `MultithreadedAnalyzeCommandBase`, raised when analyzing with the `--hashes` switch. [#2459](https://github.com/microsoft/sarif-sdk/pull/2549). There was a previous attempt to fix this in [#2447](https://github.com/microsoft/sarif-sdk/pull/2447).
-* FEATURE: Allow initialization of file regions cache in `InsertOptionalDataVisitor` (previously initialized exclusively from `FileRegionsCache.Instance`).
-* BUGFIX: Resolve issue where `match-results-forward` command fails to generate VersionControlDetails data. [#2487](https://github.com/microsoft/sarif-sdk/pull/2487)
-* BUGFIX: Remove duplicated rule definitions when executing `match-results-forward` commands for results with sub-rule ids. [#2486](https://github.com/microsoft/sarif-sdk/pull/2486)
-* BUGFIX: Update `merge` command to properly produce runs by tool and version when passed the `--merge-runs` argument. [#2488](https://github.com/microsoft/sarif-sdk/pull/2488)
-* BUGFIX: Eliminate `IOException` and `DirectoryNotFoundException` exceptions thrown by `merge` command when splitting by rule (due to invalid file characters in rule ids). [#2513](https://github.com/microsoft/sarif-sdk/pull/2513)
-* BUGFIX: Fix classes inside NotYetAutoGenerated folder missing `virtual` keyword for public methods and properties, by regenerate and manually sync the changes. [#2537](https://github.com/microsoft/sarif-sdk/pull/2537)
-* FEATURE: Allow external set of `MaxFileSizeInKilobytes`, which will allow SDK users to change the value. (Default value is 1024) [#2578](https://github.com/microsoft/sarif-sdk/pull/2578)
-* FEATURE: Add a Github validation rule `GH1007`, which requires flattened result message so GHAS code scanning can ingest the log. [#2580](https://github.com/microsoft/sarif-sdk/issues/2580)
-* BUGFIX: MSBuild Converter now accepts case insensitive keywords and supports PackageValidator msbuild log output. [#2579](https://github.com/microsoft/sarif-sdk/pull/2579)
+* BREAKING: Mark `AnalyzeCommandBase` as obsolete. This type will be removed in the next significant update.  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* BREAKING: `LogUnhandledEngineException` no longer has a return value (and updates the `RuntimeErrors` context property directly as other helpers do).  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* BUGFIX  : Resolve hangs due to unhandled exceptions during multithreaded analysis file enumeration phase.  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* BUGFIX  : Another attempt to resolve 'InvalidOperationException' with message `Collection was modified; enumeration operation may not execute` in `MultithreadedAnalyzeCommandBase`, raised when analyzing with the `--hashes` switch. [#2459](https://github.com/microsoft/sarif-sdk/pull/2549). There was a previous attempt to fix this in [#2447](https://github.com/microsoft/sarif-sdk/pull/2447).
+* BUGFIX  : Resolve issue where `match-results-forward` command fails to generate VersionControlDetails data. [#2487](https://github.com/microsoft/sarif-sdk/pull/2487)
+* BUGFIX  : Remove duplicated rule definitions when executing `match-results-forward` commands for results with sub-rule ids. [#2486](https://github.com/microsoft/sarif-sdk/pull/2486)
+* BUGFIX  : Update `merge` command to properly produce runs by tool and version when passed the `--merge-runs` argument. [#2488](https://github.com/microsoft/sarif-sdk/pull/2488)
+* BUGFIX  : Eliminate `IOException` and `DirectoryNotFoundException` exceptions thrown by `merge` command when splitting by rule (due to invalid file characters in rule ids). [#2513](https://github.com/microsoft/sarif-sdk/pull/2513)
+* BUGFIX  : Fix classes inside NotYetAutoGenerated folder missing `virtual` keyword for public methods and properties, by regenerate and manually sync the changes. [#2537](https://github.com/microsoft/sarif-sdk/pull/2537)
+* BUGFIX  : MSBuild Converter now accepts case insensitive keywords and supports PackageValidator msbuild log output. [#2579](https://github.com/microsoft/sarif-sdk/pull/2579)
 * BUGFIX: Eliminate `NullReferenceException` when file hashing fails (due to file locked or other errors reading the file). [#2596](https://github.com/microsoft/sarif-sdk/pull/2596)
+* FEATURE : Provide `PluginDriver` property (`AdditionalOptionsProvider`) that allows additional options to be exported (typically for command-line arguments).  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* FEATURE : Provide `LogFileSkippedDueToSize` that fires a warning notification if any file is skipped due to exceeding size threshold. [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* FEATURE : Provide overridable `ShouldEnqueue` predicate method to filter files from driver processing.  [#2559](https://github.com/microsoft/sarif-sdk/pull/2599)
+* FEATURE : Allow external set of `MaxFileSizeInKilobytes`, which will allow SDK users to change the value. (Default value is 1024) [#2578](https://github.com/microsoft/sarif-sdk/pull/2578)
+* FEATURE : Add a Github validation rule `GH1007`, which requires flattened result message so GHAS code scanning can ingest the log. [#2580](https://github.com/microsoft/sarif-sdk/issues/2580)
+* FEATURE : Provide mechanism to populate `SarifLogger` with a `FileRegionsCache` instance.
+* FEATURE : Allow initialization of file regions cache in `InsertOptionalDataVisitor` (previously initialized exclusively from `FileRegionsCache.Instance`).
 
 ## **v3.1.0** [Sdk](https://www.nuget.org/packages/Sarif.Sdk/3.1.0) | [Driver](https://www.nuget.org/packages/Sarif.Driver/3.1.0) | [Converters](https://www.nuget.org/packages/Sarif.Converters/3.1.0) | [Multitool](https://www.nuget.org/packages/Sarif.Multitool/3.1.0) | [Multitool Library](https://www.nuget.org/packages/Sarif.Multitool.Library/3.1.0)
 
 * BUGFIX: Loosen `System.Collections.Immutable` minimum version requirement to 1.5.0. [#2504](https://github.com/microsoft/sarif-sdk/pull/2533)
 
 ## **v3.0.0** [Sdk](https://www.nuget.org/packages/Sarif.Sdk/3.0.0) | [Driver](https://www.nuget.org/packages/Sarif.Driver/3.0.0) | [Converters](https://www.nuget.org/packages/Sarif.Converters/3.0.0) | [Multitool](https://www.nuget.org/packages/Sarif.Multitool/3.0.0) | [Multitool Library](https://www.nuget.org/packages/Sarif.Multitool.Library/3.0.0)
+
 * BUGFIX: Loosen Newtonsoft.JSON minimum version requirement to 6.0.8 (for .NET framework) or 9.0.1 (for all other compilations) for Sarif.Sdk. Sarif.Converts requires 8.0.1, minimally, for .NET framework compilations.
 * BUGFIX: Broaden set of supported .NET frameworks for compatibility reasons. Sarif.Sdk, Sarif.Driver and Sarif.WorkItems requires net461.
 

src/Sarif.Driver/Sdk/AnalyzeCommandBase.cs

@@ -12,6 +12,7 @@
 
 namespace Microsoft.CodeAnalysis.Sarif.Driver
 {
+    [Obsolete("AnalyzeCommandBase will be deprecated entirely soon. Use MultithreadedAnalyzeCommandBase instead.")]
     public abstract class AnalyzeCommandBase<TContext, TOptions> : PluginDriverCommand<TOptions>
         where TContext : IAnalysisContext, new()
         where TOptions : AnalyzeOptionsBase
@@ -97,12 +98,13 @@ public override int Run(TOptions options)
                     catch (Exception ex)
                     {
                         // These exceptions escaped our net and must be logged here
-                        RuntimeErrors |= Errors.LogUnhandledEngineException(_rootContext, ex);
+                        Errors.LogUnhandledEngineException(_rootContext, ex);
                         ExecutionException = ex;
                         return FAILURE;
                     }
                     finally
                     {
+                        RuntimeErrors |= _rootContext.RuntimeErrors;
                         logger.AnalysisStopped(RuntimeErrors);
                     }
                 }
@@ -207,7 +209,7 @@ protected virtual void ValidateOptions(TContext context, TOptions analyzeOptions
             succeeded &= ValidateFile(context, analyzeOptions.OutputFilePath, DefaultPolicyName, shouldExist: null);
             succeeded &= ValidateInvocationPropertiesToLog(context, analyzeOptions.InvocationPropertiesToLog);
             succeeded &= analyzeOptions.ValidateOutputOptions(context);
-            succeeded &= analyzeOptions.MaxFileSizeInKilobytes > 0;
+            succeeded &= context.MaxFileSizeInKilobytes >= 0;
 
             if (!succeeded)
             {
@@ -217,6 +219,18 @@ protected virtual void ValidateOptions(TContext context, TOptions analyzeOptions
             }
         }
 
+        protected virtual bool ShouldEnqueue(string file, TContext context)
+        {
+            bool shouldEnqueue = IsTargetWithinFileSizeLimit(file, context.MaxFileSizeInKilobytes, out long fileSizeInKb);
+
+            if (!shouldEnqueue)
+            {
+                Warnings.LogFileSkippedDueToSize(context, file, fileSizeInKb);
+            }
+
+            return shouldEnqueue;
+        }
+
         internal AggregatingLogger InitializeLogger(AnalyzeOptionsBase analyzeOptions)
         {
             _tool = Tool.CreateFromAssemblyData();
@@ -257,7 +271,7 @@ private ISet<string> CreateTargetsSet(TOptions analyzeOptions)
                 foreach (string file in fileSpecifier.Files)
                 {
                     // Only include files that are below the max size limit.
-                    if (IsTargetWithinFileSizeLimit(file, _rootContext.MaxFileSizeInKilobytes))
+                    if (ShouldEnqueue(file, _rootContext))
                     {
                         targets.Add(file);
                     }
@@ -288,10 +302,13 @@ protected virtual TContext CreateContext(
             {
                 Logger = logger,
                 RuntimeErrors = runtimeErrors,
-                Policy = policy
+                Policy = policy ?? new PropertiesDictionary()
             };
 
-            context.MaxFileSizeInKilobytes = options.MaxFileSizeInKilobytes;
+            context.MaxFileSizeInKilobytes =
+                options.MaxFileSizeInKilobytes >= 0
+                ? options.MaxFileSizeInKilobytes
+                : AnalyzeContextBase.MaxFileSizeInKilobytesDefaultValue;
 
             if (filePath != null)
             {

src/Sarif.Driver/Sdk/AnalyzeOptionsBase.cs

@@ -11,6 +11,17 @@ namespace Microsoft.CodeAnalysis.Sarif.Driver
     [Verb("analyze", HelpText = "Analyze one or more binary files for security and correctness issues.")]
     public abstract class AnalyzeOptionsBase : CommonOptionsBase
     {
+        public AnalyzeOptionsBase()
+        {
+            // TODO: these defaults need to be converted to the configuration
+            // property pattern as followed by MaxFileSizeInKilobytes.
+            Traces = new string[] { };
+            Kind = new List<ResultKind> { ResultKind.Fail };
+            Level = new List<FailureLevel> { FailureLevel.Warning, FailureLevel.Error };
+
+            MaxFileSizeInKilobytes = AnalyzeContextBase.MaxFileSizeInKilobytesDefaultValue;
+        }
+
         [Value(0,
                HelpText = "One or more specifiers to a file, directory, or filter pattern that resolves to one or more binaries to analyze.")]
         public IEnumerable<string> TargetFileSpecifiers { get; set; }
@@ -118,7 +129,7 @@ public abstract class AnalyzeOptionsBase : CommonOptionsBase
         [Option(
             "max-file-size-in-kb",
             HelpText = "The maximum file size (in kilobytes) that will be analyzed.",
-            Default = 1024)]
-        public int MaxFileSizeInKilobytes { get; set; } = 1024;
+            Default = -1)]
+        public long MaxFileSizeInKilobytes { get; set; }
     }
 }

src/Sarif.Driver/Sdk/ExportConfigurationCommandBase.cs

@@ -16,11 +16,17 @@ public override int Run(ExportConfigurationOptions exportOptions)
 
             try
             {
-                PropertiesDictionary allOptions = new PropertiesDictionary();
+                var allOptions = new PropertiesDictionary();
 
                 // The export command could be updated in the future to accept an arbitrary set
                 // of analyzers for which to build an options XML file suitable for configuring them.
-                ImmutableArray<IOptionsProvider> providers = CompositionUtilities.GetExports<IOptionsProvider>(RetrievePluginAssemblies(DefaultPluginAssemblies, exportOptions.PluginFilePaths));
+                var providers = new List<IOptionsProvider>(CompositionUtilities.GetExports<IOptionsProvider>(RetrievePluginAssemblies(DefaultPluginAssemblies, exportOptions.PluginFilePaths)));
+
+                if (AdditionalOptionsProvider != null)
+                {
+                    providers.Add(AdditionalOptionsProvider);
+                }
+
                 foreach (IOptionsProvider provider in providers)
                 {
                     IOption sampleOption = null;