fix missing results in flow template + feature: internal matchers using `internal: true` #4582

tarunKoyalwar · 2024-01-05T18:53:40Z

Proposed Changes

refactor flow engine to use scan.ScanContext
fix missing results/events in flow templates
add feature to hide result/event in matchers using internal: true
updated integration test to match new behaviour of flow & internal: true
fix missing extractors in file protocol
fix: dynamic values overwritten when merging results (file protocol)
closes Support for Internal Matchers using internal: true #4580
closes Missing results when iterating over protocol in flow #4581
closes Redirected response matches are lost when executed from Flow #4443
closes Nuclei Beta v3 multi protocol issues if the regex extract many result #4058

Note

Nuclei before this PR < 3.1.4 only printed/logged last event / final result when using flow with matchers but this implicit behaviour caused issue (see: #4581 ) . Hence this now needs to be done explicitly in much cleaner and controlled way by adding internal: true to matchers

tarunKoyalwar · 2024-01-05T19:03:50Z

Example Template

id: flow-multi-events

info:
  name: Flow Multiple Events
  author: pdteam
  severity: info


flow: |
  dns()
  for(var i = 0; i < 5; i++) {
    http();
  }

dns:
  - name: "{{FQDN}}"
    type: NS

    matchers:
      - type: word
        words:
          - "IN\tNS"
        internal: true

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    matchers:
      - type: status
        status:
          - 200

Nuclei

$  ./nuclei -u scanme.sh -t a.yaml                                                                                         

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.4-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.1.4-dev (development)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[flow-multi-events] [http] [info] https://scanme.sh
[flow-multi-events] [http] [info] https://scanme.sh
[flow-multi-events] [http] [info] https://scanme.sh
[flow-multi-events] [http] [info] https://scanme.sh
[flow-multi-events] [http] [info] https://scanme.sh

tarunKoyalwar · 2024-01-05T19:13:22Z

this also seems to fix #4443 , where nuclei < 3.1.4 found match was being replaced / overwritten in flow

id: flow_matcher_bug
info:
  name: Flow matcher bug
  author: Levente Kovats
  severity: medium
  description: Matches from further redirected responses are lost when executed from Flow
  reference:
      - https://docs.projectdiscovery.io/templates/protocols/http/basic-http
      - https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections

flow: http();

http:
  - method: GET
    path:
      - "{{BaseURL}}"
    redirects: true # Follow HTTP redirects
    
    matchers:
      # detects the final page that does not redirect any further
      - type: status
        status:
          - 200

    extractors:
      # export the original URL because Nuclei output the matched one, which is probably after the redirection(s)
      - type: kval
        name: original-URL
        kval:
          - host
        internal: true
      # export the status code
      - type: kval
        name: status
        kval:
          - status_code

Before this PR / fix (match found but no result event)

$ nuclei -target "https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F" -v -matcher-status -follow-redirects -t tmp.yaml -debug 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.3

		projectdiscovery.io

[INF] Current nuclei version: v3.1.3 (latest)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [flow_matcher_bug] Dumped HTTP request for https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

GET /redirect-to?url=https%3A%2F%2Fscanme.sh%2F HTTP/1.1
Host: httpbin.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/4E423F
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[DBG] [flow_matcher_bug] Dumped HTTP response https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

HTTP/1.1 200 OK
Connection: close
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Fri, 05 Jan 2024 19:11:42 GMT

ok
[DBG] [flow_matcher_bug] Dumped HTTP response https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

HTTP/1.1 302 FOUND
Connection: close
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 05 Jan 2024 19:11:41 GMT
Location: https://scanme.sh/
Server: gunicorn/19.9.0
Content-Length: 0

This PR / fix

$  ./nuclei -target "https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F" -v -matcher-status -follow-redirects -t tmp.yaml -debug

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.4-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.1.4-dev (development)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [flow_matcher_bug] Dumped HTTP request for https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

GET /redirect-to?url=https%3A%2F%2Fscanme.sh%2F HTTP/1.1
Host: httpbin.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2762.73 Safari/537.36
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[DBG] [flow_matcher_bug] Dumped HTTP response https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

HTTP/1.1 200 OK
Connection: close
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Fri, 05 Jan 2024 19:11:53 GMT

ok
[flow_matcher_bug:status-1] [matched] [http] [medium] https://scanme.sh/ [200]
[DBG] [flow_matcher_bug] Dumped HTTP response https://httpbin.org/redirect-to?url=https%3A%2F%2Fscanme.sh%2F

HTTP/1.1 302 FOUND
Connection: close
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 05 Jan 2024 19:11:52 GMT
Location: https://scanme.sh/
Server: gunicorn/19.9.0
Content-Length: 0

tarunKoyalwar · 2024-01-05T20:07:18Z

closes #4058

setup

create data.txt with example urls to simulate a codebase with urls

$ cat data.txt 
https://banlek-28e05.firebaseio.com/nuclei.json
https://byclean-1525737949784.firebaseio.com/nuclei.json
https://byclean-152573794978d.firebaseio.com/nuclei.json

Template

id: insecure-firebase-database-Beta-v3

info:
  name: Insecure Firebase Database
  author: rafaelwdornelas
  severity: high
  description: If the owner of the app have set the security rules as true for both "read" & "write" an attacker can probably dump database and write his own data to firebase database.
  reference:
    - https://blog.securitybreached.org/2020/02/04/exploiting-insecure-firebase-database-bugbounty
  metadata:
    verified: true
  tags: firebase,google,misconfig

flow: |
  file();
  for (let token of template.tokens){
    set("token", token)
    http();
  }

file:
  - extensions:
      - all

    extractors:
      - type: regex
        name: tokens
        internal: true
        regex:
          - "[a-z0-9.-]+\\.firebaseio\\.com"
          - "[a-z0-9.-]+\\.firebaseapp\\.com"

http:
  - method: GET
    path:
      -  "https://{{token}}/{{randstr}}.json"

    matchers-condition: and
    matchers:

      - type: word
        part: header
        words:
          - "application/json"

      - type: status
        status:
          - 200

$ ./nuclei -u data.txt -t tmp.yaml        

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.4-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.1.4-dev (development)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 0 URL from httpx
[insecure-firebase-database-Beta-v3] [http] [high] https://banlek-28e05.firebaseio.com/2aYA5LPmlndKvu2cITFugKurwMh.json
[insecure-firebase-database-Beta-v3] [http] [high] https://byclean-1525737949784.firebaseio.com/2aYA5LPmlndKvu2cITFugKurwMh.json

tarunKoyalwar added 3 commits January 5, 2024 20:50

log warnings + use scanContext in flow

4549901

refactor flow to use scanContext + log all events

683da2b

feat: internal matcher

71d5962

tarunKoyalwar self-assigned this Jan 5, 2024

tarunKoyalwar changed the title ~~issue flow events~~ fix missing results in flow template + feature: internal matchers using internal: true Jan 5, 2024

tarunKoyalwar marked this pull request as ready for review January 5, 2024 18:55

fix integration test

19a2675

tarunKoyalwar added 2 commits January 6, 2024 01:19

bug fix extractor: merge dynamic values, fix missing extractors in file

7b991ed

flow: fix 'No Results Found' if last statement output is false

46c5cce

fix unit test

ee5fb71

tarunKoyalwar requested review from ehsandeep and dogancanbakir January 5, 2024 20:10

ehsandeep approved these changes Jan 7, 2024

View reviewed changes

ehsandeep merged commit 02a9b86 into dev Jan 7, 2024
12 checks passed

ehsandeep deleted the issue-flow-events branch January 7, 2024 23:42

BrewTestBot mentioned this pull request Jan 8, 2024

nuclei 3.1.4 Homebrew/homebrew-core#159289

Merged

tarunKoyalwar mentioned this pull request Jan 10, 2024

How to disable printing root matcher with flow? #4613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix missing results in flow template + feature: internal matchers using `internal: true` #4582

fix missing results in flow template + feature: internal matchers using `internal: true` #4582

tarunKoyalwar commented Jan 5, 2024 •

edited

Loading

tarunKoyalwar commented Jan 5, 2024

tarunKoyalwar commented Jan 5, 2024

tarunKoyalwar commented Jan 5, 2024

fix missing results in flow template + feature: internal matchers using internal: true #4582

fix missing results in flow template + feature: internal matchers using internal: true #4582

Conversation

tarunKoyalwar commented Jan 5, 2024 • edited Loading

Proposed Changes

tarunKoyalwar commented Jan 5, 2024

Example Template

Nuclei

tarunKoyalwar commented Jan 5, 2024

Before this PR / fix (match found but no result event)

This PR / fix

tarunKoyalwar commented Jan 5, 2024

closes #4058

setup

Template

fix missing results in flow template + feature: internal matchers using `internal: true` #4582

fix missing results in flow template + feature: internal matchers using `internal: true` #4582

tarunKoyalwar commented Jan 5, 2024 •

edited

Loading