Make it easier to filter output

[tpetchel]

Consider the scenario where you have a large number of compliance failures (for example, from the CIS benchmarks) and you want to tackle a few at a time.

With local remediation through Test Kitchen, you can scan temporary instances directly against a full compliance profile. kitchen verify potentially produces a huge volume of output, which can be difficult to wade through.

In the Learn Chef tutorials (CentOS) (Ubuntu), we have the user grep the output of kitchen verify to find the errors they care about.

What makes this difficult is that you need to grep for the matcher code itself. For example:

$ kitchen verify > verify.txt 2>&1
$ cat verify.txt | grep -A 10 '\d*).*should match /^Status: active$/'

It can be painful running grep because you need to escape the matcher code itself (because you're grepping for a regex from a regex tool.)

A few ways that might make this easier are to:

1. Add the name of the rule to the output (for example, Set SSH Protocol to 2). This would make gripping easier. If the user is scaring against multiple profiles, perhaps also include the profile name as a way to namespace things (perhaps overkill.)

2. Add a way for the user to specify in their .kitchen.yml file which rules they care about. Then scan against only those rules. For example:

   verifier:
     name: inspec
     inspec_tests:
       - compliance://cis/cis-centos7-level1
     filter_rules:
       - 'Set SSH Protocol to 2'
       - 'Ensure Firewall is active'
   

3. Enable the user to create a .inspec.yml file (or similar) that defines specific rules to include or exclude, similar to how .rubocop.yml works.

These options would produce much leaner output when you run kitchen verify, and would eliminate the need to redirect the output to a file and then process that file.

Those are just a few ideas that came to mind. Perhaps there are better alternatives.

[chris-rock]

@tpetchel I agree, that we need to improve our reporting!

[adamleff]

Considering all the reporting and formatting happens in InSpec core, this issue is probably better logged there instead of here in kitchen-inspec even if a future enhancement to kitchen-inspec is desired to support filtering. I would be 👎 on that given that it would be inconsistent behavior across different verifier plugins, but we can cross that bridge at a later time.

This likely will need changes to the existing formatter, or perhaps a new formatter that consolidates profile/test/result into a single line for easier searching. Regardless, I've opened an enhancement request in inspec/inspec#2166 and will close this issue.