Continuous Fuzzing

[yevgenypats]

Hi, I'm Yevgeny Pats Founder of Fuzzit - Continuous fuzzing as a service platform.

We have a free plan for OSS and I would be happy to contribute a PR if that's interesting.
The PR will include the following

• go-fuzz fuzzers
• Continuous Fuzzing of master branch which will generate new corpus and look for new crashes
• Regression on every PR that will run the fuzzers through all the generated corpus and fixed crashes from previous step. This will prevent new or old bugs from crippling into master.

You can see our basic example here and you can see an example of "in the wild" integration here.

Let me know if this is something worth working on.

Also, we have a reward program. If you are interested in implementing the fuzzers and the integration yourself I’ll be happy to reward you as well as to get unbiased feedback on how smooth the integration was.

Cheers,
Yevgeny

[mholt]

That would actually be great. I've been meaning to put Caddy through more fuzzing.

Unfortunately I don't have time to do the integration myself in the near future but I'd welcome and review a PR if you'd like to contribute it!

I think the main areas for fuzzing would be the Caddyfile parser and of course the HTTP requests and TLS handshakes.

We are switching to Caddy 2 fairly soon though (within the year) -- will it require a lot of work to bring the fuzzers into the v2 code, you think? (See the v2 branch. Still undergoing active development.)

[mohammed90]

I'd love to work on the fuzzing for Caddy 2! Incidentally, I was looking at the Caddy 2 branch last night looking for areas to fuzz and couldn't find anything fuzz-able, mostly because the configuration has shifted from Caddyfile to JSON. Perhaps there will be one when the Caddyfile adapter lands? I've fuzzed the Caddyfile parser before (#2096 ) and you've fixed about all the crashes. The ones left out are inconsequential.
Can you point me at other stuff to fuzz? Wouldn't fuzzing HTTP requests and TLS handshake belong to the stdlib fuzzing rather than Caddy?

[mholt]

@mohammed90 That would be great! Yes, I remember your PR from before, would love to have you contribute again. Consider letting @yevgenypats know if you want to participate in their rewards program.

Perhaps there will be one when the Caddyfile adapter lands?

Definitely will. It's the open PR, #2699 -- but I am currently making some biiig changes to it so maybe don't fuzz it quite yet.

Can you point me at other stuff to fuzz? Wouldn't fuzzing HTTP requests and TLS handshake belong to the stdlib fuzzing rather than Caddy?

Hmm, I think I mean more about fuzzing the ServeHTTP methods using requests filled with fuzz data. Similarly, fuzzing Caddy's GetCertificate or GetConfigForClient TLS handshake functions. The standard lib calls these. What do you think?

Caddy 2 also has a few API endpoints for administration. Those might be worth looking into as well.

Oh, our v2 replacer might also be worth fuzzing; I just optimized it pretty heavily and am hoping it will perform well even with naughty inputs: #2674. The replacer is what allows you to use variables in the config.

[yevgenypats]

hi @mohammed90 , If you can contribute this it will be awesome and of-course you will be rewarded:) once you have the fuzz targets ready and you get to integrate them with Fuzzit feel free to join our slack and ask any questions/feature-requests/feedback regarding the integration.

[mohammed90]

Great. Let me fiddle around with the codebase to come up with the Fuzz funcs. I have created a scratch azure-pipelines.yml file based on the .travis.yml sample files I've seen, so that's easy and 90% out of the way. Figuring what the Fuzz funcs should do is the tricky bit.

[mohammed90]

@mholt are you or Gitter, or any other preferred messaging platform? I need to understand a few areas in the codebase to figure out how to fuzz them best. Also, we need Fuzzit API key encrypted by Azule Pipelines and set as a secret there to include in azure-pipelines.yml and test out the script.

[mholt]

We have a public forum here: https://caddy.community - it updates in real-time so it can be used like a chat, and the conversation can be preserved so it will be useful for future searchers.

Or if we need to discuss anything privately or one-on-one, we have a dev slack I can invite you to. Just let me know your email address. (I typically prefer the forum)

[mholt]

@mohammed90 The v2 Caddyfile adapter has been merged in, so that is a candidate for fuzzing I think. It's in the v2 branch and the docs are updated with information about how to use the Caddyfile! (It's still a little rough around the edges but the bigger picture will stay the same.)

[mohammed90]

Great, I'll rebase the few changes I have. The dev slack sounds good to me. I prefer discussions in smaller circles to have smaller noise rate and higher possibility of meaningful responses. Should I shoot an email to the email address that's on your website to securely transport my email address?

[mholt]

Well, no email is really secure :) If you're that concerned about it, DM me on the forums with your email instead.

[mohammed90]

Just trying to keep my email address away from crawlers looking for them :) Anyways, I got this error on the forum "Sorry, you cannot send a personal message to that user."

[mholt]

Huh. Okay, I've temporarily changed the default setting so that "trust level 0" users can send private messages. Try again, sorry :)

[mohammed90]

Done! Thanks

[mholt]

It appears the rewards are no longer available. :-/

Fuzzing is mostly integrated though, thanks for all your hard work @mohammed90 !