Automated fuzzing on Fuzzit

[bookmoons]

Integrates with Fuzzit for automated fuzzing. Proposed in #2670.

Under this setup Travis runs fuzzing on every build. It runs a fuzz regression test locally of past crashes and submits to Fuzzit for a full fuzzing run. The build ends as soon as the fuzzing run is in the queue. Fuzzit emails when it finds crashes.

This tries to fuzz the suggested targets plus some other identified parsers. Targets are:

• control-api-http - HTTP requests to the control API server.
• control-api-tcp - Binary requests to the control API server.
• parse-config - Fuzz the config parser.
• parse-reference - Fuzz the Docker image reference parser.

There's a JDWP specification parser in pkg/skaffold/debug that seems suitable for fuzzing. It's tough to get at because it's a private function. If you think it's worth fuzzing exposing it would make it possible.

A skaffold org is preconfigured on Fuzzit. If whoever will be administering signs in to create an account, I can get you added to the org.

Thank you for considering.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[bookmoons]

@googlebot signed it

[googlebot]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[codecov]

Codecov Report

Merging #2835 into master will increase coverage by 1.78%.
The diff coverage is 55.23%.

Impacted Files	Coverage Δ
cmd/skaffold/app/cmd/run.go	47.05% <ø> (-5.58%)	⬇️
cmd/skaffold/app/cmd/flags.go	100% <ø> (ø)	⬆️
cmd/skaffold/app/cmd/completion.go	14.28% <0%> (+1.78%)	⬆️
cmd/skaffold/app/cmd/credits/export.go	0% <0%> (ø)
cmd/skaffold/app/cmd/schema/list.go	100% <100%> (ø)
cmd/skaffold/app/cmd/build.go	82.5% <100%> (ø)	⬆️
cmd/skaffold/app/cmd/debug.go	50% <100%> (ø)	⬆️
cmd/skaffold/app/cmd/deploy.go	80.64% <100%> (+2.07%)	⬆️
cmd/skaffold/app/cmd/render.go	42.1% <100%> (ø)	⬆️
cmd/skaffold/app/flags/build_output.go	82.85% <100%> (+4.28%)	⬆️
... and 130 more

[bookmoons]

@dominikh Hi, I built on your good work for this PR. The googlebot is asking for consent to have your commits merged. Would you mind looking at it?

[bookmoons]

I see there's a strange error in the build. A type error even though the types are identical.

/home/travis/gopath/src/skaffold/pkg/skaffold/docker/auth.go:69:39: cannot use cf (variable of type *configfile.ConfigFile) as *configfile.ConfigFile value in argument to gcp.AutoConfigureGCRCredentialHelper

Not sure what that's about, it works locally. Will be looking into this.

[dominikh]

@googlebot I consent.

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[bookmoons]

Thanks @dominikh.

[bookmoons]

Applied the Go modules approach from dvyukov/go-fuzz#195 (comment) to get this working. Thanks @mvdan for the pattern.

This is ready for review.

[balopat]

This looks good but I'm hesitant to add 8 more minutes to our feedback loop. Can you please change it to a job instead of a stage so at least it can run in parallel with the other jobs?

[bookmoons]

Will do, thanks for looking at it.

[bookmoons]

Rebased this and converted the fuzz stages to jobs. I'll check back to see how the build goes.

If you'd like to sign in to Fuzzit to make an account I can add you to the org. Then final setup is like this:

• In Fuzzit, switch to the skaffold org with the dropdown at the top.
• In settings grab an API key.
• In repo settings in Travis paste it to envvar FUZZIT_API_KEY.

[dgageot]

@balopat do you think we should merge this (when it's rebased)?

[balopat]

I'm not convinced. One thing is the slowness of the test, the other is that most of the test code seems to be HTTP specific instead of Skaffold's test code. It feels like this kind of code should live in a fuzz-http library or similar...I'm not sure I want to maintain this.

[dgageot]

@balopat I agree. I'm sorry @bookmoons but I think maintaining that would cost us more that it brings.

[balopat]

Sounds good. Apologies for dragging this out this long. Closing.