BUG: scanning gitlab private repositories

[adam-moss]

Describe the bug
scorecard current errors out when attempting to scan private GitLab projects even though it has an access token that gives access.

Reproduction steps
Steps to reproduce the behaviour:

1. Create a private project in GitLab
2. Set GITLAB_AUTH_TOKEN to a suitable token with access to the project
3. Run scorecard --repo https://gitlab.com/path/to/my/project

Expected behaviour
I expected scorecard would successfully scan the project and provide the results in the normal way.

Additional context
Merge request #3208 added an explicitly check for private projects in https://github.com/ossf/scorecard/blob/main/clients/gitlabrepo/client.go#L65, presumably to optimise the cron based schedule scans. However as it is implemented as a global ban rather than a cron specific option it prevents all scanning of private projects.

[gabibguti]

Hey @adam-moss! Can you provide some more information? Are you using the latest version of Scorecard (v4.13.0)? Did you grant access to api and read_api scopes for the GitLab token? And, can you share the error log?

[adam-moss]

Hi,

Yes, latest per

./scorecard: OpenSSF Scorecard

GitVersion:    4.13.0
GitCommit:     e1d3abc7fd2bdfe8819ac19b5c82815ea20890e6
GitTreeState:  clean
BuildDate:     2023-10-05T18:58:53Z
GoVersion:     go1.21.2
Compiler:      gc
Platform:      darwin/amd64

An example would be:

%  scorecard --repo https://gitlab.com/adam-moss/danger-bot
Starting [Dangerous-Workflow]
Starting [License]
Starting [Dependency-Update-Tool]
Starting [Fuzzing]
Starting [Contributors]
Starting [Code-Review]
Starting [Packaging]
Starting [Token-Permissions]
Starting [Vulnerabilities]
Starting [SAST]
Starting [CII-Best-Practices]
Starting [Pinned-Dependencies]
Starting [Maintained]
Starting [Security-Policy]
Starting [Binary-Artifacts]
Starting [CI-Tests]
Starting [Branch-Protection]
Starting [Signed-Releases]
Error: RunScorecard: repo unreachable: repo inaccessible: adam-moss/danger-bot access level private
2023/10/18 21:26:03 error during command execution: RunScorecard: repo unreachable: repo inaccessible: adam-moss/danger-bot access level private

Compared with a public repo, e.g.:

%  scorecard --repo https://gitlab.com/adam-moss/dotfiles
Starting [Security-Policy]
Starting [Dangerous-Workflow]
Starting [Binary-Artifacts]
Starting [SAST]
Starting [Contributors]
Starting [Fuzzing]
Starting [Branch-Protection]
Starting [CII-Best-Practices]
Starting [Packaging]
Starting [Dependency-Update-Tool]
Starting [Signed-Releases]
Starting [Token-Permissions]
Starting [Pinned-Dependencies]
Starting [Maintained]
Starting [CI-Tests]
Starting [License]
Starting [Vulnerabilities]
Starting [Code-Review]
Finished [Security-Policy]
Finished [Dangerous-Workflow]
Finished [Binary-Artifacts]
Finished [SAST]
Finished [Contributors]
Finished [Fuzzing]
Finished [Branch-Protection]
Finished [CII-Best-Practices]
Finished [Packaging]
Finished [Dependency-Update-Tool]
Finished [Signed-Releases]
Finished [Token-Permissions]
Finished [Pinned-Dependencies]
Finished [Maintained]
Finished [CI-Tests]
Finished [License]
Finished [Vulnerabilities]
Finished [Code-Review]

RESULTS
-------
...

Thanks 😄

[spencerschrock]

Your private repo example gives me a 404, which is what I would expect as a third party:

go run main.go --repo https://gitlab.com/adam-moss/danger-bot --format json | jq
Error: RunScorecard: repo unreachable: adam-moss/danger-bot     GET https://gitlab.com/api/v4/projects/adam-moss/danger-bot: 404 {message: 404 Project Not Found}
2023/10/18 14:03:23 error during command execution: RunScorecard: repo unreachable: adam-moss/danger-bot        GET https://gitlab.com/api/v4/projects/adam-moss/danger-bot: 404 {message: 404 Project Not Found}
exit status 1

I've created a test private repo and was able to access it with my PAT (I'm running at HEAD, but identical results at v4.13.0):

go run main.go --repo https://gitlab.com/spencerschrock/private-test --format json | jq
{
  "date": "2023-10-18T14:05:00-07:00",
  "repo": {
    "name": "gitlab.com/spencerschrock/private-test",
    "commit": "382e5a59c6a7f93c3ca8cf43ca82afa704a7ce8f"
  },

My token is as follows:

Token name(Click to sort ascending)	Scopes(Click to sort ascending)	Created(Click to sort ascending)	Last Used The last time a token was used(Click to sort ascending)	Expires(Click to sort ascending)	Action
scorecard	read_api, read_user	Jul 27, 2023	5 minutes ago	in 9 months

[gabibguti]

Seems like were falling under different errors tho.

scorecard/clients/gitlabrepo/client.go

Lines 86 to 92 in 63fff3c

	if err != nil {
	return sce.WithMessage(sce.ErrRepoUnreachable, proj+"\t"+err.Error())
	}
	if err = checkRepoInaccessible(repo); err != nil {
	return sce.WithMessage(sce.ErrRepoUnreachable, err.Error())
	}

When repoducing I'm also falling in the first IF, but seems like adam is falling in the second IF.

[gabibguti]

It's a bug on GitLab for newly created projects. The setting repository_access_level should come as private but comes as enabled. Changing the repo visibility to public then private again "fixes" the setting, and now the error is reproducible.
https://gitlab.com/gitlab-org/gitlab/-/issues/345858

[gabibguti]

I believe we should change the logic to: "if is a private repo but has a token, then it's fine".

[spencerschrock]

It's a bug on GitLab for newly created projects. The setting repository_access_level should come as private but comes as enabled. Changing the repo visibility to public then private again "fixes" the setting, and now the error is reproducible. https://gitlab.com/gitlab-org/gitlab/-/issues/345858

Nice find!

My vote is to defer until @raghavkaul is available, to confirm intent behind the original patch.

[gabibguti]

Agreed.

Meanwhile I will share some other findings:
InitRepo already fails if you're looking for a private project as "not found" if you don't have any token that makes the project visible. That means, if the project is private we should not need this condition in checkRepoInaccessible. If you do have a token that makes the private project visible, then you should be able to run the scores without stopping in checkRepoInaccessible.

But, yeah, Raghav to take a look at cases we want to avoid in the cronjob, if it's really private access, or only disabled access. Fun thing is that private access is not necessarily of private repos.

Project visibility: one of public, private, internal (deprecated)
Repository visibility: one of enabled, disabled with one of only project members, everyone

public + repository disabled => disabled
public + repository enabled + everyone => enabled
public + repository enabled + only project members => private
private + repository disabled => disabled
private + repository enabled + everyone => X (cannot exist)
private + repository enabled + only project members => private

[raghavkaul]

Thanks @gabibguti for looking into it, good find! The reason for the original patch was to make cron errors easier to debug/easier to prune private repos from the cron.

According to GitLab docs:

If you disable Repository, project users cannot access:

Merge requests
CI/CD
Git Large File Storage
Packages

It's hard to run Scorecard without access that info. So I think the DisabledAccessControl part of checkRepoInaccessible would still make sense to raise an error, and I agree with @spencerschrock that the PrivateAccessControl check should be combined with whether GITLAB_AUTH_TOKEN is valid for that repo (or, at least, whether GITLAB_AUTH_TOKEN has been set at all?)

[gabibguti]

As discussed, we will consider only DisabledAccessControl case in checkRepoInaccessible, and let PrivateAccessControl cases continue the workflow.