Add JSON lines output

[ocervell]

This adds JSON lines support to cariddi, to match with similar tools (ffuf, gau, gospider, etc...).

Usage:

cariddi -json -s -e -ext 7 -err -info

Example output:

{
  "url": "http://localhost:8000/scans/?id=3",
  "method": "GET",
  "status_code": 200,
  "words": 35,
  "lines": 1,
  "content_type": "application/json",
  "content_length": 340,
  "matches": {
    "filetype": {},
    "parameters": [
      {
        "name": "id",
        "attacks": [
          "SQLi",
          "XSS"
        ]
      }
    ],
    "errors": [
      {
        "name": "PHP error",
        "match": "php error"
      },
      {
        "name": "PHP error",
        "match": "failed to open stream"
      },
      {
        "name": "PHP error",
        "match": "fatal error"
      },
      {
        "name": "General error",
        "match": "fatal error"
      },
      {
        "name": "MySQL error",
        "match": "MySQLSyntaxErrorException"
      }
    ],
    "infos": [
      {
        "name": "Email address",
        "match": "test@gmail.com"
      },
      {
        "name": "Internal IP address",
        "match": "192.168.1.18"
      },
      {
        "name": "IPv4 address",
        "match": "192.168.1.18"
      }
    ],
    "secrets": [
      {
        "name": "AWS Access Key",
        "match": "AKIAJSIE27KKMHXI3BJQ"
      }
    ]
  }
}

Some other changes, so that the output is parseable with jq, such as cariddi -json | jq '.':

• Banner header is output to stderr instead of stdout (common in other tools that support JSON lines output such as httpx)
• Omit CLI output if -json is active: only JSON lines should be returned, no additional text

Fixes:

• Add JSON functionality and console reporting of secrets, errors #103

[edoardottt]

Hi @ocervell ! Thank you so much for the PR, really appreciated!

I will review the changes when there'll be no errors and zero conflicts (and some free time oc 😭, ask anything)

[ocervell]

@edoardottt Thanks ! Should be good now, updated with devel branch, if you can re-run the tests that would be amazing ;)

[edoardottt]

@edoardottt Thanks ! Should be good now, updated with devel branch, if you can re-run the tests that would be amazing ;)

The linter is showing some errors, I use golangci-lint to maintain a good code level. If you need help ask me :)

[ocervell]

@edoardottt should be ready for review / merge ;)

BTW, would be nice to update the README for install steps, for me only this worked from the main directory:

go get -u -d ./...
go run cmd/cariddi/main.go -help

Simply running go get tells me go: no package to get in current directory

[edoardottt]

Sorry Olivier but I'm writing the thesis these days and I'm super busy. Whenever I find enough time to review the code i will get back to u, I promise it won't take so much. If you want in the meanwhile you can push some test cases to test the functions added (for example GetJSONString with fake colly responses, secrets and so on to see if all the checks are implemented... fields or headers missing, weird data, edge case scenarios.. whatever in your mind).

Regarding install steps..

1. Which Go version you have installed in your env?
2. Did you try the snap method (if you are in a Linux/Ubuntu env) ?

[ocervell]

Try running:

echo <URL> | go run cmd/cariddi/main.go -json -ext 7 -err -info -s | jq '.'

and appreciate beautiful JSON output ;)

[edoardottt]

lgtm!

[edoardottt]

Thanks @ocervell for your amazing work. really appreciated. I'm going to open new issues as I have found some problems to be solved (my fault aha)
If you want to open new issues and work on new features or fix some bugs, just go ahead :)