Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow configuring HTTP proxy with Trivy scanner #84

Closed
larssb opened this issue Jul 9, 2020 · 13 comments · Fixed by #154
Closed

Allow configuring HTTP proxy with Trivy scanner #84

larssb opened this issue Jul 9, 2020 · 13 comments · Fixed by #154
Assignees
Labels
🚀 enhancement New feature or request

Comments

@larssb
Copy link

larssb commented Jul 9, 2020

ISSUE

starboard find vulns deployment/some_deployment --namespace default -v 3
I0708 14:43:53.094626    3804 scanner.go:55] Getting Pod template for workload: {Deployment core-scanandpay-failover-v2 default}
I0708 14:43:53.117105    3804 scanner.go:70] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0708 14:43:53.118220    3804 runner.go:79] Running task and waiting forever
I0708 14:43:53.118288    3804 runnable_job.go:47] Creating runnable job: starboard/2e7c4482-df93-4057-a5b1-dbc1a18ce353
I0708 14:43:53.137715    3804 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0708 14:43:53.138252    3804 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0708 14:43:55.586801    3804 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0708 14:43:55.587514    3804 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0708 14:43:55.723818    3804 manager.go:177] Container 2e7c4482-df93-4057-a5b1-dbc1a18ce353 terminated with Error: 2020-07-08T12:43:54.563Z                                                                       INFO     Need to update DB
2020-07-08T12:43:54.564Z        INFO    Downloading DB...
2020-07-08T12:43:54.730Z        FATAL   failed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: Get https://api.github.com/repos/aquasecurity/trivy-db/releases: x509: certificate signed by unknown authority
error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

TECHNICAL DETAILS

  • K8S version: 1.14.6
  • starboard version: 0.2.5
  • K8S runs on a VMWare platform on-prem
    • Behind a corporate FW and proxy.
  • kubectl version: 1.17.3
  • Running it in Windows Subsystem for Linux v1. Using the Ubuntu v18.04 distro

EXPECTATIONS

To be able to execute starboard find vulns deployment/some_deployment without this issue

DEBUGGING - TRIED

  • Tried using: --insecure-skip-tls-verify the issue remained
  • Tried removing the certificate-authority-data sections from the kubeconfig file. The issue remained
    • Tried with and without the --insecure-skip-tls-verify parameter. Same issue

OTHER

I'm posting this issue in this repo as the challenge seems to be related to specifically trivy-db. As that is being called via starboard.

I created the same issue on the trivy-db repo as well. But was asked to create it here as well. See the issue here.

Looking forward to any tips and pointers. Let me know if more info is needed.

Thank you very much.

@danielpacak
Copy link
Contributor

danielpacak commented Jul 9, 2020

Thank you for reporting this @larssb . I did check on my end and cannot reproduce it anymore. It looks like a temporary issue with GitHub release page from which Trivy downloads the Trivy DB artifact.

Could you confirm that you still have the same problem?

@larssb
Copy link
Author

larssb commented Jul 9, 2020

Bummer 👎 ... :-) I still get the same error. It could easily be something on my side. Network wise. Company proxy or the like. However, pointers on how-to troubleshoot this would be highly regarded.

Thank you

@larssb
Copy link
Author

larssb commented Jul 9, 2020

Tried with a cntlm proxy. To ensure proper authentication when required by our company proxy. This works with git, when required. But for starboard the issue remains.
There is some accepted MITM going on. Still a couple of tips on troubleshooting this would be highly appreciated.

@danielpacak
Copy link
Contributor

I'd first make sure that you can download Trivy DB from GitHub and run maybe Trivy directly, without Starboard:

@larssb
Copy link
Author

larssb commented Jul 10, 2020

Hi @danielpacak.

  • I can download the trivy.db file with the curl cmd in WSL
  • Running trivy throws
2020-07-10T10:53:24.923+0200    DEBUG   Updating database metadata...
2020-07-10T10:53:24.934+0200    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-07-10 00:22:52.309793773 +0000 UTC, NextUpdate: 2020-07-10 12:22:52.309793473 +0000 UTC
2020-07-10T10:53:28.132+0200    DEBUG   Vulnerability type:  [os library]
panic: invalid page type: 0: 4
goroutine 1 [running]:
go.etcd.io/bbolt.(*Cursor).search(0xc0005d4f38, 0xc0005d503c, 0x4, 0x4, 0x4)
        /go/pkg/mod/go.etcd.io/bbolt@v1.3.4/cursor.go:250 +0x353
go.etcd.io/bbolt.(*Cursor).seek(0xc0005d4f38, 0xc0005d503c, 0x4, 0x4, 0x510, 0x8, 0xc000086700, 0x7f33b5f31460, 0x0, 0x7f33b3cbbe90, ...)
        /go/pkg/mod/go.etcd.io/bbolt@v1.3.4/cursor.go:159 +0x7d
go.etcd.io/bbolt.(*Bucket).Bucket(0xc0003ec1d8, 0xc0005d503c, 0x4, 0x4, 0x8)
        /go/pkg/mod/go.etcd.io/bbolt@v1.3.4/bucket.go:105 +0xd4
go.etcd.io/bbolt.(*Tx).Bucket(...)
        /go/pkg/mod/go.etcd.io/bbolt@v1.3.4/tx.go:102
github.com/aquasecurity/fanal/cache.FSCache.MissingBlobs.func1(0xc0003ec1c0, 0x0, 0xc0003ec1c0)
        /go/pkg/mod/github.com/aquasecurity/fanal@v0.0.0-20200528202907-79693bf4a058/cache/cache.go:185 +0x87
go.etcd.io/bbolt.(*DB).View(0xc000139800, 0xc0005d5270, 0x0, 0x0)
        /go/pkg/mod/go.etcd.io/bbolt@v1.3.4/db.go:725 +0xa8
github.com/aquasecurity/fanal/cache.FSCache.MissingBlobs(0xc000139800, 0xc0003de330, 0x22, 0xc00054c3c0, 0x47, 0xc000030ac0, 0x3, 0x4, 0xc0005d57f8, 0x4e, ...)
        /go/pkg/mod/github.com/aquasecurity/fanal@v0.0.0-20200528202907-79693bf4a058/cache/cache.go:184 +0xfe
github.com/aquasecurity/fanal/artifact/image.Artifact.Inspect(0x7fffc050d8be, 0xa, 0x1338100, 0xc000218ac0, 0x1326ce0, 0xc0000f6120, 0x132b0e0, 0xc0000360f0, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/aquasecurity/fanal@v0.0.0-20200528202907-79693bf4a058/artifact/image/image.go:44 +0x306
github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact(0x1310c00, 0xc0003ab770, 0x1310a80, 0xc0003ab710, 0x132b0e0, 0xc0000360f0, 0xc00040d760, 0x2, 0x2, 0x0, ...)
        /home/circleci/project/pkg/scanner/scan.go:85 +0x66
github.com/aquasecurity/trivy/internal/artifact.run(0xc000430240, 0xc00000ea78, 0x12eae8c, 0x5, 0x100, 0xc00041e020, 0x1c, 0x0, 0x0, 0x1bf08eb000, ...)
        /home/circleci/project/internal/artifact/run.go:81 +0x8d5
github.com/aquasecurity/trivy/internal/artifact.ImageRun(0xc000430240, 0x12, 0x24)
        /home/circleci/project/internal/artifact/image.go:49 +0x1c7
github.com/urfave/cli/v2.(*Command).Run(0xc0003cd440, 0xc0003c9bc0, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/command.go:164 +0x4b9
github.com/urfave/cli/v2.(*App).RunContext(0xc000001800, 0x132b0e0, 0xc0000360f0, 0xc000030100, 0x4, 0x4, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:306 +0x790
github.com/urfave/cli/v2.(*App).Run(...)
        /go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:215
main.main()
        /home/circleci/project/cmd/trivy/main.go:18 +0x87

Apparently because of this issue

I'll see if I can get around the WSL issue somehow. I'll get back to this.

@larssb
Copy link
Author

larssb commented Jul 14, 2020

Circumvented the WSL issue for a moment by using starboard on Windows directly. Exactly the same error I get.

@larssb
Copy link
Author

larssb commented Jul 14, 2020

Using trivy directly on Windows. It's the same error:

C:\WORK\Apps\starboard> docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /WORK/Temp/cache:/root/.cache/ aquasec/trivy coop/core-scanandpay-buyflowtest
Unable to find image 'aquasec/trivy:latest' locally
latest: Pulling from aquasec/trivy
df20fa9351a1: Pull complete
e6af5451f500: Pull complete
297f0ed972f8: Pull complete
51a6785a0a2f: Pull complete
2a60dc89834f: Pull complete
Digest: sha256:f37b2b23b0e92da0edf1c99e999549bfdaf01660a5a39b4bc507ec7bba5d9a9c
Status: Downloaded newer image for aquasec/trivy:latest
2020-07-14T13:00:14.781Z        WARN    You should avoid using the :latest tag as it is cached. You need to specify '--clear-cache' option when :latest image is changed
2020-07-14T13:00:14.783Z        INFO    Need to update DB
2020-07-14T13:00:14.783Z        INFO    Downloading DB...
2020-07-14T13:00:14.975Z        FATAL   failed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: Get https://api.github.com/repos/aquasecurity/trivy-db/releases: x509: certificate signed by unknown authority

Any tips for pin-pointing the culprit here?

Thank you @danielpacak

@danielpacak
Copy link
Contributor

LTM as some networking issue. I see that you're running behind a corporate proxy. Have you tried setting HTTP_PROXY or HTTPS_PROXY envs?

First we need to resolve the issue of running standalone trivy in your env:

$ HTTPS_PROXY=yourproxy trivy alpine:3.10.2

@danielpacak
Copy link
Contributor

Also make sure that you can download trivy-db artifacts with curl or wget:

$ curl -LO https://github.com/aquasecurity/trivy-db/releases/download/v1-2020071412/trivy-light.db.gz

@larssb
Copy link
Author

larssb commented Jul 15, 2020

Thank you @danielpacak

I've know tried out your suggestions. I can use trivy directly by sending/setting an environment variable with the -e parameter to the docker run... cmd.
The variable I set is of course == HTTPS_PROXY=.....

So as I see it. The question that remains is. How can one specify the proxy that trivy should use when starboard is used. Executing starboard help and/or starboard find help does not reveal, at least as I see it, how it is possible to specify the proxy server to use.

Thank you.

@danielpacak
Copy link
Contributor

Starboard does not support specifying HTTP(S) proxy for Trivy. We have to implement that. It's somehow related to general topic of how we configure Starboard and other scanners. Let me think about it and I'll comment in this thread or create a follow up issue.

@larssb
Copy link
Author

larssb commented Jul 15, 2020

Sure @danielpacak .. I'll happily let you think about this. Sounds good. I'll use trivy for now directly. And then see and go from there. It's likely not going to be an issue using starboard on our CI/CD platform. However, it would really be wonderful and useful to be able to use starboard locally.

Thank you and looking forward to your thoughts.

@danielpacak
Copy link
Contributor

danielpacak commented Sep 18, 2020

@larssb We have recently introduced a generic configuration map for Starboard CLI, which is created on init. So far it was only used by Polaris, but now we can use it to configure scanners by adding well known keys to the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/managed-by: starboard
  name: starboard
  namespace: starboard
data:
  trivy.httpProxy: http://somehost:8080
  trivy.severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
  # ...

So the workflow would be:

  1. Initialize Starboard and create the starboard ConfigMap with the default settings, i.e. no HTTP(S) proxy for Trivy:
    $ kubectl starboard init
    
  2. Set HTTP(S) proxy for Trivy:
    $ kubectl patch configmap starboard -n starboard \
      --type merge \
      -p '{"data": {"trivy.httpProxy":"http://proxy-host:8080"}}'
    
  3. Scan Deployment for vulnerabilities using Trivy:
    $ kubectl starbaord find vulnerabilities deployment/my-deployment
    

NB I renamed this issue to reflect the latest status and the problem that we're trying to solve

danielpacak added a commit that referenced this issue Sep 18, 2020
Some people might run their (dev) clusters behind the proxy.
It is possible to set the HTTP_PROXY environment variable
when using Trivy directly. This commit makes it possible
to use Starboard CLI and pass proxy confing to Trivy
by setting the trivy.httpProxy configuration parameter
before you run the starboard find vulnerabilities command.

$ starboard init
$ kubectl patch configmap starboard -n starboard \
  --type merge \
  -p '{"data": {"trivy.httpProxy":"http://your-proxy:9001"}}'
$ starboard find vulnerabilities deploy/my-deployment

Resolves: #84

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
danielpacak added a commit that referenced this issue Sep 18, 2020
Some people might run their (dev) clusters behind the proxy.
It is possible to set the HTTP_PROXY environment variable
when using Trivy directly. This commit makes it possible
to use Starboard CLI and pass HTTP proxy config to Trivy by
setting the trivy.httpProxy configuration parameter before
you run the starboard find vulnerabilities command.

$ starboard init
$ kubectl patch configmap starboard -n starboard \
  --type merge \
  -p '{"data": {"trivy.httpProxy":"http://your-proxy:9001"}}'
$ starboard find vulnerabilities deploy/my-deployment

Resolves: #84

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
@danielpacak danielpacak changed the title starboard find vulns deployment/some_deployment throws certificate signed by unknown authority Allow configuring HTTP proxy with Trivy scanner Sep 18, 2020
@danielpacak danielpacak self-assigned this Sep 18, 2020
@danielpacak danielpacak added the 🚀 enhancement New feature or request label Sep 18, 2020
@danielpacak danielpacak added this to the Release v0.4.0 milestone Sep 18, 2020
danielpacak added a commit that referenced this issue Sep 18, 2020
Some people might run their (dev) clusters behind the proxy.
It is possible to set the HTTP_PROXY environment variable
when using Trivy directly. This commit makes it possible
to use Starboard CLI and pass HTTP proxy config to Trivy by
setting the trivy.httpProxy configuration parameter before
you run the starboard find vulnerabilities command.

$ starboard init
$ kubectl patch configmap starboard -n starboard \
  --type merge \
  -p '{"data": {"trivy.httpProxy":"http://your-proxy:9001"}}'
$ starboard find vulnerabilities deploy/my-deployment

Resolves: #84

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
danielpacak added a commit that referenced this issue Sep 19, 2020
Some people might run their (dev) clusters behind the proxy.
It is possible to set the HTTP_PROXY environment variable
when using Trivy directly. This commit makes it possible
to use Starboard CLI and pass HTTP proxy config to Trivy by
setting the trivy.httpProxy configuration parameter before
you run the starboard find vulnerabilities command.

$ starboard init
$ kubectl patch configmap starboard -n starboard \
  --type merge \
  -p '{"data": {"trivy.httpProxy":"http://your-proxy:9001"}}'
$ starboard find vulnerabilities deploy/my-deployment

Resolves: #84

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚀 enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants