Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzzing layer enhancements + input-types support #4477

Merged
merged 92 commits into from
Mar 13, 2024

Conversation

Ice3man543
Copy link
Member

@Ice3man543 Ice3man543 commented Dec 9, 2023

Proposed changes

Supersedes #4163

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@Ice3man543
Copy link
Member Author

Ice3man543 commented Dec 9, 2023

Needs tests related to fuzzing layer execution + other points noted by tarun in #4163

Known Issues / Planned Changes

  • Add more params etc that shouldn't be fuzzed by default (ex. __viewstate, etc)
  • (common/fuzz) Rewrite querypartRule & headersPartRule tests with new methods/logic
  • (core/inputs/formats) Replace http.ReadRequest with manual/custom parser as it is strict http request parser and does not support unsafe raw requests
  • (core/inputs/formats) use OrderedParam to maintain Order of Headers while parsing (currently we use map[string][]string)
  • (core/inputs/formats) fix failing input format test of openapi, json and raw request
  • move package (pkg/protocols/common/fuzz) -> (pkg/fuzz) [ to avoid deep nested directories and easier access to fuzz module)

@tarunKoyalwar tarunKoyalwar self-assigned this Feb 7, 2024
@tarunKoyalwar
Copy link
Member

tarunKoyalwar commented Feb 13, 2024

Checklist

  • Complete refactor to use pkg/inputs/types (RequestResponse) [ Response part is a placeholder for now ]
  • Unified Target/Input Provider ( consolidate MetaInput / RawRequest )
  • Add New Yaml Input Format for Http Target data ( generated by proxify)
  • Refactor to use Unified/New Input Provider Interface
  • Remove -input-file flag and consolidate its logic using -l flag
  • Add -fuzz flag to load/execute fuzz protocol templates
  • Missing SDK Options for new flags
  • Rename Part names of (url , response)
  • Add support for replace-regex rule type to allow custom replacement
  • Fix Rule applicability, it only considers url and does not consider entire http request
  • Unit Tests for Each Part ( query,path,header,body)
  • Integration Test for Each Part (query,path,header,body)
  • Rewrite Integration Test using fuzzing-playground instead of router
  • Add support for new multipart/form-data body format
  • Introduce Static Auth From Yaml File using -secrets
    • Pass this to postman,openapi,swagger (to generate requests on fly)
    • Allow specifying variables set/update in postman,openapi,swagger from -var flag (WIP)
    • Implement logic for passing creds via ScanContext in tmplexec
    • Yaml Format ( Regex / Exact Match Auth Support )
    • 1password integration ( will be implemented in followup release )
  • Setup up Automation to generate and update default exclude list of headers and cookies ( a good source for this is wappalyzer project which contains headers and cookies of some technologies ) (see: Setup Automation to update default ignore list of headers,cookies for fuzzing #4843)
  • Follow-up issue to implement chunked Provider (eliminate hmap) (Scan Planner #4808 (comment))

Integration Tests Added

Existing fuzz integration test only use url, keeping those aside we are adding Integration tests for all components using fuzz playground ^

[✓] Test "fuzz/fuzz-query-num-replace.yaml" passed!
[✓] Test "fuzz/fuzz-header-ssrf.yaml" passed!
[✓] Test "fuzz/fuzz-path-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-cookie-error-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-body-json-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-body-multipart-form-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-body-params-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-body-xml-sqli.yaml" passed!
[✓] Test "fuzz/fuzz-body-generic-sqli.yaml" passed!

@tarunKoyalwar tarunKoyalwar marked this pull request as draft February 13, 2024 21:54
@tarunKoyalwar
Copy link
Member

tarunKoyalwar commented Mar 10, 2024

New Changes

Improved Loggging when required vars are missing

$ ./nuclei -l openapi.yaml -im openapi -t a.yaml -V X-PDCP-Key=123456 -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.0-dev

		projectdiscovery.io

[VER] openapi: skipping optional param (limit) in (query) in request [GET] /api/v1/cves due to missing value (limit)
[VER] openapi: skipping optional param (cpe.cpe) in (query) in request [GET] /api/v1/cves due to missing value (cpe.cpe)
[ERR] openapi: Found 1 missing parameters, use -skip-format-validation flag to skip requests or update missing parameters generated in required_openapi_params.yaml file,you can also specify these vars using -var flag in (key=value) format
[VER] openapi: missing params: [id]

auto generate / load required openapi variables file in CLI Mode

$ cat required_openapi_params.yaml                                                          
var:
    - id=

    # Optional parameters
    # - cpe.cpe=
    # - limit=

Skipping format validation will use placeholders for known datatypes

 $ ./nuclei -l openapi.yaml -im openapi -t a.yaml -V X-PDCP-Key=123456 -v -sfv

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.0-dev

		projectdiscovery.io

[VER] Started metrics server at localhost:9092
[INF] Current nuclei version: v3.2.0-dev (development)
[INF] Current nuclei-templates version: v9.7.7 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 82
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 5
[VER] [host-header-injection] Sent HTTP request to https://cve.projectdiscovery.io/api/v1/filters
[VER] [host-header-injection] Sent HTTP request to https://cve.projectdiscovery.io/api/v1/cpes/cpe:2.3:a:atlassian:confluence_server:*:*:*:*:*:*:*:*?product=jira&vendor=atlassian
[VER] [host-header-injection] Sent HTTP request to https://cve.projectdiscovery.io/api/v1/cves?assignee=psirt%40adobe.com&cpe.cpe=string&cpe.framework=wordpress&cpe.product=acrobat_dc&cpe.vendor=adobe&cve_description=Adobe+Acrobat+and+Reader+versions+2019.010.20069+and+earlier%2C+2019.010.20069+and+earlier%2C+2017.011.30113+and+earlier+version%2C+and+2015.006.30464+and+earlier+have+an+use+after+free+vulnerability.+Successful+exploitation+could+lead+to+arbitrary+code+execution+.&cve_id=CVE-2019-7070&cvss_metrics.cvss2.score=9.3&cvss_metrics.cvss2.severity=critical&cvss_metrics.cvss2.vector=CVSS%3A2.0%2FAV%3AN%2FAC%3AM%2FAu%3AN%2FC%3AC%2FI%3AC%2FA%3AC&cvss_metrics.cvss30.score=8.8&cvss_metrics.cvss30.severity=high&cvss_metrics.cvss30.vector=CVSS%3A3.0%2FAV%3AN%2FAC%3AL%2FPR%3AN%2FUI%3AR%2FS%3AU%2FC%3AH%2FI%3AH%2FA%3AH&cvss_metrics.cvss31.score=7&cvss_metrics.cvss31.severity=high&cvss_metrics.cvss31.vector=CVSS%3A3.1%2FAV%3AL%2FAC%3AH%2FPR%3AL%2FUI%3AN%2FS%3AU%2FC%3AH%2FI%3AH%2FA%3AH&cvss_score=8.8&epss.epss_percentile=0.80053&epss.epss_score=0.00826&fields=cve_id%2Ccve_description&hackerone.count=0&is_exploited=true&is_oss=false&is_poc=true&is_remote=true&is_template=false&kev.added_date=2022-04-15T00%3A00%3A00Z&kev.due_date=2022-05-06T00%3A00%3A00Z&limit=0&nuclei_templates.created_at=2020-04-05T23%3A31%3A09%2B05%3A30&nuclei_templates.template_issue=https%3A%2F%2Fgh.neting.cc%2Fprojectdiscovery%2Fnuclei-templates%2Fissues%2F7549&nuclei_templates.template_issue_type=mention&nuclei_templates.template_path=http%2Fcves%2F2019%2FCVE-2019-12314.yaml&nuclei_templates.template_pr=https%3A%2F%2Fgh.neting.cc%2Fprojectdiscovery%2Fnuclei-templates%2Fpull%2F3200&nuclei_templates.template_url=https%3A%2F%2Fcloud.projectdiscovery.io%2Fpublic%2FCVE-2019-12314&nuclei_templates.updated_at=2023-12-29T09%3A30%3A44Z&offset=0&oss.created_at=2009-05-21+01%3A33%3A45+%2B0000+UTC&oss.description=Mirror+of+Apache+ActiveMQ&oss.forks=1407&oss.language=Java&oss.pushed_at=2023-12-12+17%3A51%3A19+%2B0000+UTC&oss.stars=2221&oss.subscribers=200&oss.topics.%24_has=php&oss.updated_at=2023-12-29+09%3A29%3A55+%2B0000+UTC&oss.url=https%3A%2F%2Fgh.neting.cc%2Fapache%2Factivemq&patch_url.%24_has=https%3A%2F%2Fhelpx.adobe.com%2Fsecurity%2Fproducts%2Facrobat%2Fapsb19-07.html&poc.%24.added_at=2019-04-02T12%3A50%3A46Z&poc.%24.source=trickest&poc.%24.url=https%3A%2F%2Fmedium.com%2F%40alt3kx%2Fa-reflected-xss-in-print-archive-system-v2015-release-2-6-cve-2019-10685-b60763b7768b&published_at=2019-05-24T19%3A29%3A02.080&reference.%24_has=https%3A%2F%2Fwww.zerodayinitiative.com%2Fadvisories%2FZDI-19-210%2F&severity=high&shodan.count=150&shodan.query.%24_has=cpe%3A%22cpe%3A2.3%3Aa%3Aadobe%3Acoldfusion%22&sort_asc=age_in_days%2Chackerone.rank&sort_desc=cvss_score&updated_at=2019-08-21T16%3A20%3A31.353&vendor_advisory=https%3A%2F%2Fhelpx.adobe.com%2Fsecurity%2Fproducts%2Facrobat%2Fapsb19-07.html&vuln_status=confirmed&vulnerable_cpe.%24_has=cpe%3A2.3%3Aa%3Aatlassian%3Ajira%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A&weaknesses.%24.cwe_id=CWE-416&weaknesses.%24.cwe_name=Use+After+Free
[VER] [host-header-injection] Sent HTTP request to https://cve.projectdiscovery.io/api/v1/cves/search
[VER] [host-header-injection] Sent HTTP request to https://cve.projectdiscovery.io/api/v1/cve/string
[INF] No results found. Better luck next time!

Copy link
Member

@ehsandeep ehsandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge conflict

tarunKoyalwar and others added 5 commits March 11, 2024 04:37
* fix tag include logic

* fix unit test

* remove quoting in extractor output

* remove quote in debug code command
* feat: issue tracker URLs in JSON + misc fixes

* misc changes

* feat: status update support for issues

* feat: report metadata generation hook support

* feat: added CLI summary of tickets created

* misc changes
* introduce `disable-unsigned-templates` flag

* minor

* skip instead of exit

* remove duplicate imports

* use stats package + misc enhancements

* force display warning + adjust skipped stats in unsigned count

* include unsigned skipped templates without -dut flag

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* purge cache on global callback set

* lint

* purging cache

* purge cache in runner after loading templates

* include internal cache from parsers + add global cache register/purge via config

* remove disable cache purge option

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
@tarunKoyalwar tarunKoyalwar requested a review from ehsandeep March 10, 2024 23:16
@ehsandeep ehsandeep merged commit fa56800 into dev Mar 13, 2024
12 checks passed
@ehsandeep ehsandeep deleted the fuzzing-layer-enhancements branch March 13, 2024 21:38
@tarunKoyalwar tarunKoyalwar linked an issue Mar 14, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment