Support multiple output format

[dogancanbakir]

This PR adds support for outputting in all formats. Closes #1150.

CLI flag

-oA, -output-all                    filename to write output results in all formats

CLI commands for test:
This will output in all formats:

go run httpx.go -u scanme.sh -o test -oA

This will output in plain text format:

go run httpx.go -u scanme.sh -o test

This will output in JSON format:

go run httpx.go -u scanme.sh -o test -json

This will output in CSV format:

go run httpx.go -u scanme.sh -o test -csv

[ehsandeep]

• not working

go run httpx.go -u scanme.sh -o test

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

        projectdiscovery.io

[INF] Current httpx version v1.3.2 (outdated)

• this should still output default plain text output

go run httpx.go -u scanme.sh -o test -oA

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

		projectdiscovery.io

[INF] Current httpx version v1.3.2 (outdated)
timestamp,asn,-,csp,tls,hash,extract_regex,cdn_name,port,-,url,input,location,title,scheme,error,webserver,body,content_type,method,host,path,favicon,favicon_path,final_url,header,raw_header,request,time,jarm,chain_status_codes,a,cname,tech,extracts,chain,words,lines,status_code,content_length,failed,vhost,websocket,cdn,http2,pipeline,headless_body,screenshot_bytes,stored_response_path,screenshot_path
{"timestamp":"2023-06-28T17:24:04.389127+05:30","hash":{"body_md5":"444bcb3a3fcf8389296c49467f27e1d6","body_mmh3":"-2088429648","body_sha256":"2689367b205c16ce32ed4200942b8b8b1e262dfc70d9bc9fbc77c49699a4f1df","body_simhash":"590632390773907579","header_md5":"48439a0e86962c1162569138235ff2fb","header_mmh3":"-718729990","header_sha256":"21a934cf46e8168e412e48f6c8ff615dc8d417620424808fe580a05a78d4498d","header_simhash":"9814104581732024175"},"port":"443","url":"https://scanme.sh","input":"scanme.sh","scheme":"https","content_type":"text/plain","method":"GET","host":"128.199.158.128","path":"/","time":"322.952333ms","a":["128.199.158.128","2400:6180:0:d0::91:1001"],"words":1,"lines":1,"status_code":200,"content_length":2,"failed":false}
2023-06-28 17:24:04.389127 +0530 IST m=+0.668319542,<nil>,<nil>,<nil>,<nil>,map[body_md5:444bcb3a3fcf8389296c49467f27e1d6 body_mmh3:-2088429648 body_sha256:2689367b205c16ce32ed4200942b8b8b1e262dfc70d9bc9fbc77c49699a4f1df body_simhash:590632390773907579 header_md5:48439a0e86962c1162569138235ff2fb header_mmh3:-718729990 header_sha256:21a934cf46e8168e412e48f6c8ff615dc8d417620424808fe580a05a78d4498d header_simhash:9814104581732024175],[],,443,"HTTP/1.1 200 OK
Connection: close
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Wed, 28 Jun 2023 11:54:04 GMT

ok",https://scanme.sh,scanme.sh,,,https,,,,text/plain,GET,128.199.158.128,/,,,,map[],,,322.952333ms,,[],[128.199.158.128 2400:6180:0:d0::91:1001],[],[],map[],[],1,1,200,2,false,false,false,false,false,false,,[],,

• not working

chaos -d hackerone.com -silent | go run httpx.go

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

        projectdiscovery.io

[INF] Current httpx version v1.3.2 (outdated)

• build fail

Running test cases for "http"
[✘] Test "Raw request with non standard rfc path via stdin" failed: incorrect number of results 
[✘] Test "Standard HTTP GET Request with match response time" failed: incorrect number of results 
[✘] Test "Output Filter Condition" failed: incorrect number of results 
[✘] Test "Standard HTTPS GET Request" failed: incorrect number of results 
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.(*issue480).Execute(0xc000759de0?)
    /home/runner/work/httpx/httpx/cmd/integration-test/http.go:298 +0x226
main.main()
    /home/runner/work/httpx/httpx/cmd/integration-test/integration-test.go:36 +0x511

[ehsandeep]

Not working:

• echo scanme.sh | ./httpx -csv
• echo scanme.sh | ./httpx -json

Unexpected behavior:

• -oA option adds hashes to the default output (it should follow the same behavior as default)

echo scanme.sh | ./httpx -oA -silent
https://scanme.sh [444bcb3a3fcf8389296c49467f27e1d6,-2088429648,2689367b205c16ce32ed4200942b8b8b1e262dfc70d9bc9fbc77c49699a4f1df,590632390773907579]

Additions:

• Add validation for -oA option to be used only when -output option is specified.
• Add tests for the new feature being implemented.

[Mzack9999]

• Minor changes (import + potential helper function)
• Merge conflict. I think the final merge could be this, but I catch the occasion to get your confirmation:

if options.OutputAll && options.Output == "" {
		gologger.Fatal().Msg("Please specify an output file using -o/-output when using -oA/-output-all")
	}

	if options.OutputAll {
		options.JSONOutput = true
		options.CSVOutput = true
	}

	if cfgFile != "" {
		if !fileutil.FileExists(cfgFile) {
			gologger.Fatal().Msgf("given config file '%s' does not exist", cfgFile)
		}
		// merge config file with flags
		if err := flagSet.MergeConfigFile(cfgFile); err != nil {
			gologger.Fatal().Msgf("Could not read config: %s\n", err)
		}
	}

[dogancanbakir]

@Mzack9999 You are correct about the conflict.

[Mzack9999]

• Suggesting a different approach to handle stdout with -oa
• To confirm: -o with -json or -csv adds by default .json and .csv respectively. Is this the wanted behavior @ehsandeep ?

[ehsandeep]

• Suggesting a different approach to handle stdout with -oa
• To confirm: -o with -json or -csv adds by default .json and .csv respectively. Is this the wanted behavior @ehsandeep ?

No, the extension should auto appended only for -oa option. cc @dogancanbakir

echo google.com | httpx -json -o test will create test
echo google.com | httpx -csv -o test will create test
echo google.com | httpx -o test will create test
echo google.com | httpx -oa -o test will create test, test.json,test.csv

[Mzack9999]

• When both -csv and -json are used I think we should also add the proper extension, actually data in both formats is written to the same file

$ echo google.com | go run . -json -o test -csv

• Integration test failing

[Mzack9999]

lgtm!

$  echo google.com | ./httpx -oA -o test
...
https://google.com
$ ls


    Directory: C:\Users\user\go\src\github.com\projectdiscovery\httpx\cmd\httpx


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        26/05/2023     14:42           1727 httpx.go
-a----        10/07/2023     14:18             19 test
-a----        10/07/2023     14:18           2195 test.csv
-a----        10/07/2023     14:18            903 test.json