Reproducible crash with core

[joshzelonis]

Apologies, I don't have experience developing with Go to take this any farther, but happy to send you the core if needed.

twitter.com.probe.gz

$ uname -a
Linux juniper 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ GOTRACEBACK=crash httpx -silent -l twitter.com.probe
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x6ff90c]

goroutine 38 [running]:
panic(0x757340, 0xa79ed0)
/usr/local/go/src/runtime/panic.go:722 +0x2c2 fp=0xc000045c30 sp=0xc000045ba0 pc=0x42e6d2
runtime.panicmem(...)
/usr/local/go/src/runtime/panic.go:199
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:394 +0x3ec fp=0xc000045c60 sp=0xc000045c30 pc=0x442ffc
github.com/projectdiscovery/httpx/common/httpx.(*HTTPX).NewRequest(0xc000012c00, 0x7c08db, 0x3, 0xc000d342e0, 0x18, 0xc000d342e0, 0x18, 0x0)
/home/josh/go/pkg/mod/github.com/projectdiscovery/httpx@v0.0.1/common/httpx/httpx.go:151 +0x7c fp=0xc000045d00 sp=0xc000045c60 pc=0x6ff90c
main.analyze(0xc000012c00, 0x7c1784, 0x5, 0xc00001c3f0, 0x10, 0x0, 0xc0000730e0, 0xc0000681e0)
/home/josh/go/pkg/mod/github.com/projectdiscovery/httpx@v0.0.1/cmd/httpx/httpx.go:165 +0x17c fp=0xc000045f18 sp=0xc000045d00 pc=0x704dcc
main.main.func2(0xc00000e100, 0xc000012c00, 0x7c1784, 0x5, 0xc0000730e0, 0xc0000681e0, 0xc00001c3f0, 0x10)
/home/josh/go/pkg/mod/github.com/projectdiscovery/httpx@v0.0.1/cmd/httpx/httpx.go:121 +0xbc fp=0xc000045fa0 sp=0xc000045f18 pc=0x706d1c
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000045fa8 sp=0xc000045fa0 pc=0x45ab51
created by main.main
/home/josh/go/pkg/mod/github.com/projectdiscovery/httpx@v0.0.1/cmd/httpx/httpx.go:119 +0x87a

[ehsandeep]

Hey @joshzelonis,

Thank a lot for filing this issue, even I'm not able to reproduce this on our end, but it might be an issue for a few cases.

root@b0x:~/test# cat twitter.com.probe | wc 
2224

root@b0x:~/test# cat twitter.com.probe | sort -u | wc 
885

root@b0x:~/test# cat twitter.com.probe | sort -u | httpx -silent | wc
667

before getting this fixed, can you help us to confirm few things, yesterday we fixed similar issue here #2

so can you update the codebase with following and try to reproduce this again?

GO111MODULE=on go get -u -v github.com/projectdiscovery/httpx/cmd/httpx@master

Tip:- you can always sort your input to reduce the extra time processing.

[joshzelonis]

Upgrading version fixed it, sorry for the duplicate.

Agree on sort, I'd removed the code around the command to try to isolate the issue but could have avoided uploading the full output from dnsprobe. 👍

[ehsandeep]

Thank you for confirming this @joshzelonis