Parts of security scorecard doesn't allow excluding issues from scoring when they have no affect on an end-user

[yrusskih]

Describe the bug
We have a low security score at https://securityscorecards.dev/viewer/?uri=github.com/JetBrains/intellij-community for Binary-Artifacts and Pinned-Dependencies categories although most of the listed paths are from ../testData/.. directories thus can't influence .

Reproduction steps
Steps to reproduce the behavior:

1. Expand Binary-Artifacts section.
2. Check paths to the files - most of them are in testData folders which doesn't affect a distribution or an end-user.
3. Expand Pinned-Dependencies section.
4. Check paths to the files - most of them are in testData folders which doesn't affect a distribution or an end-user.
5. Expand Vulnerabilities section.
6. Check paths to files caused Vulnerabilities findings - most of them are in testData folders which affects neither a distribution nor an end-user. Also most of them are related to Python dependencies although the product itself is predominantly written in Java and Python code is used there for tests.

Expected behavior
There is a way either to exclude paths not affecting an end-user or a product's distribution from the analysis, or to exclude them from scoring so a reader could see issues with a vendor's comment but in a muted form, to have a score reflecting security of an end-user product. Currently, it indicates security of files included in a repo which may have lax connection to security of a final artifact.

[spencerschrock]

Being able to declare testdata directories (along with other annotations) is something we're currently working on with maintainer annotations (Duplicate of #1907), so stay tuned.

You can see part of this in an open PR (#3905)

scorecard/config/annotations.go

Lines 20 to 23 in a36843e

	const (
	// TestData is to annotate when a check or probe is targeting a danger
	// in files or code snippets only used for test or example purposes.
	TestData Reason = "test-data"

[yrusskih]

@spencerschrock thank yoy for sharing! I've subscribed to the PR.

What's your forecast on the timeline of having this feature available for us?

[spencerschrock]

It is one of our items currently being worked on, but I don't have a great timeline, especially when seeing the results in our API.

Note: we currently exclude testdata directories (which is the Go naming convention for test data directories), but our comparison is case-sensitive. Seeing as your directory is named testData, it may be a simpler fix to use a case-insensitive comparison which would ignore your testData directory regardless of the maintainer annotation feature status.

[oreillymj]

It would be nice to exclude ad-hoc folders. We are being dinged for not having pinned dependencies in a script which sets up our ci environment. These scripts are not part of the shipped product.
https://securityscorecards.dev/viewer/?uri=github.com/intel/userspace-cni-network-plugin
Having a .ossfignore file containing a set of tests to exclude for that folder would be very useful.

Specifically, in my case, if I look at the details of what the pinned dependency check is flagging ( among others) is every line containing kubectl which is pinned to the specific version we pull down, rather than latest. https://github.com/intel/userspace-cni-network-plugin/blob/364834664f9c0434ac524a829f40905ac64592fc/ci/ci.sh#L31