Not all checks returned for repo

[mtcolman]

Describe the bug
Hi, this might not be a bug, but I didn't spot documentation that gave me the answer - apologies if I've missed something.

Reproduction steps
Steps to reproduce the behavior:

1. run curl command (with jq formatting output), see results

$ curl -s https://api.securityscorecards.dev/projects/github.com/websocket-client/websocket-client |   jq -r '. as $input | "Repo Nam
e: \($input.repo.name)", "Score: \($input.score)", (.checks[] | "\(.name) Score: \(.score), Reason: \(.reason)")'

Repo Name: github.com/websocket-client/websocket-client
Score: 5.4
Maintained Score: 10, Reason: 12 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review Score: 4, Reason: found 18 unreviewed changesets out of 30 -- score normalized to 4
CII-Best-Practices Score: 0, Reason: no effort to earn an OpenSSF best practices badge detected
License Score: 10, Reason: license file detected
Dangerous-Workflow Score: 10, Reason: no dangerous workflow patterns detected
Packaging Score: -1, Reason: packaging workflow not detected
Token-Permissions Score: 0, Reason: detected GitHub workflow tokens with excessive permissions
Binary-Artifacts Score: 10, Reason: no binaries found in the repo
Pinned-Dependencies Score: 0, Reason: dependency not pinned by hash detected -- score normalized to 0
Fuzzing Score: 10, Reason: project is fuzzed
Security-Policy Score: 0, Reason: security policy file not detected
Vulnerabilities Score: 10, Reason: 0 existing vulnerabilities detected
Signed-Releases Score: -1, Reason: no releases found
Branch-Protection Score: 0, Reason: branch protection not enabled on development/release branches
SAST Score: 0, Reason: SAST tool is not run on all commits -- score normalized to 0

15 check results returned

$ curl -s https://api.securityscorecards.dev/projects/github.com/ossf/scorecard |   jq -r '. as $input | "Repo Name: \($input.repo.na
me)", "Score: \($input.score)", (.checks[] | "\(.name) Score: \(.score), Reason: \(.reason)")'

Repo Name: github.com/ossf/scorecard
Score: 9.7
Binary-Artifacts Score: 10, Reason: no binaries found in the repo
Branch-Protection Score: -1, Reason: internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
CI-Tests Score: 10, Reason: 30 out of 30 merged PRs checked by a CI test -- score normalized to 10
CII-Best-Practices Score: 5, Reason: badge detected: passing
Code-Review Score: 10, Reason: all changesets reviewed
Contributors Score: 10, Reason: 50 different organizations found -- score normalized to 10
Dangerous-Workflow Score: 10, Reason: no dangerous workflow patterns detected
Dependency-Update-Tool Score: 10, Reason: update tool detected
Fuzzing Score: 10, Reason: project is fuzzed
License Score: 10, Reason: license file detected
Maintained Score: 10, Reason: 30 commit(s) out of 30 and 0 issue activity out of 30 found in the last 90 days -- score normalized to 10
Packaging Score: 10, Reason: publishing workflow detected
Pinned-Dependencies Score: 9, Reason: dependency not pinned by hash detected -- score normalized to 9
SAST Score: 10, Reason: SAST tool is run on all commits
Security-Policy Score: 10, Reason: security policy file detected
Signed-Releases Score: -1, Reason: no releases found
Token-Permissions Score: 10, Reason: GitHub workflow tokens follow principle of least privilege
Vulnerabilities Score: 9, Reason: 1 existing vulnerabilities detected

18 check results returned

Expected behavior
This page says "There are currently 18 checks made across 3 themes: holistic security practises, source code risk assessment and build process risk assessment." I guess I would expect the full set of checks (18) to be returned, even if I was told it the repo hadn't been assessed against this check?

[spencerschrock]

This is intentional, although perhaps not documented well.
For a longer explanation, see #3438 (comment)

[mtcolman]

Thanks @spencerschrock. Wondering if maybe an explanation could be added here or here?

Also, it would be good to have it here - not sure where the source is?

What do you think?

[spencerschrock]

Thanks @spencerschrock. Wondering if maybe an explanation could be added here or here?

Hmm, to me this is more something specific to the API, not the tool. So I would expect it to be under that part of the README.

scorecard/README.md

Lines 157 to 165 in fbffff1

	### Scorecard REST API
	To query pre-calculated scores of OSS projects, use the [REST API](https://api.securityscorecards.dev).
	To enable your project to be available on the REST API, set
	[`publish_results: true`](https://github.com/ossf/scorecard-action/blob/dd5015aaf9688596b0e6d11e7f24fff566aa366b/action.yaml#L35)
	in the Scorecard GitHub Action setting.
	Data provided by the REST API is licensed under the [CDLA Permissive 2.0](https://cdla.dev/permissive-2-0).

Also, it would be good to have it here - not sure where the source is?

The website is in another repo:
https://github.com/ossf/scorecard-webapp/blob/main/scorecards-site/content/home.md

[mtcolman]

Hi @spencerschrock just to confirm I've understood this correctly:

If we make a request to the API and there aren’t GitHub action published results available, it’ll use the weekly cron. However, in the weekly cron, CI-tests, contributors, dependency-update-tool, webhooks are disabled, so wouldn’t be returned.

[spencerschrock]

Hi @spencerschrock just to confirm I've understood this correctly:

If we make a request to the API and there aren’t GitHub action published results available, it’ll use the weekly cron. However, in the weekly cron, CI-tests, contributors, dependency-update-tool, webhooks are disabled, so wouldn’t be returned.

Correct. With the caveat that the repo might not be in the weekly cron either, in which case you would 404

[gabibguti]

Additionally to adding this explanation on the README.md (Scorecard REST API section), we could also have it on the securityscorecards.dev/viewer page too, I have faced this question a few times and I was wondering if I could add the missing checks or not.

[pnacht]

+1 to adding ? for any missing checks in the viewer.

[cpanato]

Is this still need help?

[afmarcum]

Yes, @cpanato
This is a documentation change to better explain which checks would be seen per running locally, on the cron, or scorecard-action. Displaying the information in a table may be the most clear.