fix: data race at protocolstate, contextargs & some outdated test cases

[dwisiswant0]

Proposed changes

Fixes #5815 #5819 #5833

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[dwisiswant0]

Test:

$ go test -v -run "^Test(MultiProto|Flow)With" ./pkg/tmplexec/...
?   	github.com/projectdiscovery/nuclei/v3/pkg/tmplexec	[no test files]
?   	github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/flow/builtin	[no test files]
?   	github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/generic	[no test files]
=== RUN   TestFlowWithProtoPrefix
--- PASS: TestFlowWithProtoPrefix (0.08s)
=== RUN   TestFlowWithConditionNegative
--- PASS: TestFlowWithConditionNegative (0.02s)
=== RUN   TestFlowWithConditionPositive
    flow_executor_test.go:144:
        	Error Trace:	/home/dw1/Development/PD/nuclei/pkg/tmplexec/flow/flow_executor_test.go:144
        	Error:      	Expected nil, but got: &errorutil.enrichedError{errString:"got following errors while executing flow <- [:RUNTIME] failed to execute dns:1 protocol <- could not resolve, max retries exceeded", StackTrace:"", Tags:[]string(nil), Level:0x2, OnError:(errorutil.ErrCallback)(nil)}
        	Test:       	TestFlowWithConditionPositive
        	Messages:   	could not execute template
--- FAIL: TestFlowWithConditionPositive (0.05s)
=== RUN   TestFlowWithNoMatchers
    flow_executor_test.go:165:
        	Error Trace:	/home/dw1/Development/PD/nuclei/pkg/tmplexec/flow/flow_executor_test.go:165
        	Error:      	Expected nil, but got: &errorutil.enrichedError{errString:"got following errors while executing flow <- [:RUNTIME] failed to execute dns:1 protocol <- could not resolve, max retries exceeded", StackTrace:"", Tags:[]string(nil), Level:0x2, OnError:(errorutil.ErrCallback)(nil)}
        	Test:       	TestFlowWithNoMatchers
        	Messages:   	could not execute template
--- FAIL: TestFlowWithNoMatchers (0.05s)
FAIL
FAIL	github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/flow	0.262s
=== RUN   TestMultiProtoWithDynamicExtractor
--- PASS: TestMultiProtoWithDynamicExtractor (0.11s)
=== RUN   TestMultiProtoWithProtoPrefix
--- PASS: TestMultiProtoWithProtoPrefix (5.75s)
PASS
ok  	github.com/projectdiscovery/nuclei/v3/pkg/tmplexec/multiproto	(cached)
FAIL

Tried adjusting InternalResolversList, Timeout, and Retries opts, but still no luck.

git --no-pager diff pkg/testutils/testutils.go
diff --git a/pkg/testutils/testutils.go b/pkg/testutils/testutils.go
index a1ed685c..681bdef1 100644
--- a/pkg/testutils/testutils.go
+++ b/pkg/testutils/testutils.go
@@ -54,8 +54,8 @@ var DefaultOptions = &types.Options{
 	MetricsPort:                0,
 	BulkSize:                   25,
 	TemplateThreads:            10,
-	Timeout:                    5,
-	Retries:                    1,
+	Timeout:                    30,
+	Retries:                    5,
 	RateLimit:                  150,
 	RateLimitDuration:          time.Second,
 	ProbeConcurrency:           50,
@@ -76,6 +76,7 @@ var DefaultOptions = &types.Options{
 	InteractionsPollDuration:   5,
 	GitHubTemplateRepo:         []string{},
 	GitHubToken:                "",
+	InternalResolversList:      []string{"tcp:1.1.1.1:53"},
 }

[tarunKoyalwar]

data race in memgardian

$ go test -race -v ./...                       
?   	github.com/projectdiscovery/nuclei/v3/lib	[no test files]
# github.com/projectdiscovery/nuclei/v3/lib/tests.test
ld: warning: '/private/var/folders/fs/50l_djns0db4w30ttbd6czgr0000gn/T/go-link-2964062660/000024.o' has malformed LC_DYSYMTAB, expected 98 undefined symbols to start at index 1626, found 95 undefined symbols starting at index 1626
=== RUN   TestSimpleNuclei
[INF] Your current nuclei-templates v10.0.3 are outdated. Latest is v10.0.4
[INF] Successfully updated nuclei-templates (v10.0.4) to /Users/tarun/nuclei-templates. GoodLuck!
[INF] Templates clustered: 14 (Reduced 10 Requests)
[caa-fingerprint] scanme.sh
[nameserver-fingerprint] scanme.sh
{"duration":"0:00:01","errors":"1","hosts":"1","matched":"2","percent":"94","requests":"16","rps":"15","startedAt":"2024-11-18T15:35:45.18144+05:30","templates":"24","total":"17"}
--- PASS: TestSimpleNuclei (27.54s)
=== RUN   TestSimpleNucleiRemote
[nameserver-fingerprint] scanme.sh
--- PASS: TestSimpleNucleiRemote (3.21s)
=== RUN   TestThreadSafeNuclei
==================
WARNING: DATA RACE
Write at 0x000106f1b918 by goroutine 9731:
  github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate.StartActiveMemGuardian()
      /Users/tarun/Codebase2/nuclei/pkg/protocols/common/protocolstate/memguardian.go:26 +0x64
  github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate.Init()
      /Users/tarun/Codebase2/nuclei/pkg/protocols/common/protocolstate/state.go:162 +0x1114
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x1ec
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:657 +0x138
  fmt.(*ss).doScanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:1230 +0x328
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x130
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:643 +0x74
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_util.go:52 +0x1d0
  github.com/syndtr/goleveldb/leveldb.openDB()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:130 +0x674
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x130
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:643 +0x74
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x130
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:643 +0x74
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x1ec
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:657 +0x138
  fmt.(*ss).doScanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:1230 +0x328
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x130
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:643 +0x74
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:476 +0x84
  github.com/syndtr/goleveldb/leveldb.openDB()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:125 +0x668
  github.com/syndtr/goleveldb/leveldb.Open()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:197 +0x274
  fmt.Fscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:143 +0x94
  fmt.Sscanf()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/fmt/scan.go:114 +0x130
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:643 +0x74
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/storage/file_storage.go:458 +0x364
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/session.go:113 +0xb4
  runtime.deferreturn()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/runtime/panic.go:602 +0x5c
  github.com/syndtr/goleveldb/leveldb.Open()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:183 +0xb0
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /Users/tarun/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:219 +0x6c
  github.com/projectdiscovery/hmap/store/disk.OpenLevelDB()
      /Users/tarun/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.65/store/disk/leveldb.go:25 +0x90
  github.com/projectdiscovery/hmap/store/hybrid.New()
      /Users/tarun/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.65/store/hybrid/hybrid.go:157 +0x65c
  github.com/projectdiscovery/fastdialer/fastdialer.NewDialer()
      /Users/tarun/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.2.9/fastdialer/dialer.go:87 +0x260
  github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate.Init()
      /Users/tarun/Codebase2/nuclei/pkg/protocols/common/protocolstate/state.go:151 +0x10a0
  github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit.Init()
      /Users/tarun/Codebase2/nuclei/pkg/protocols/common/protocolinit/init.go:16 +0x2c
  github.com/projectdiscovery/nuclei/v3/lib.(*NucleiEngine).init.func1()
      /Users/tarun/Codebase2/nuclei/lib/sdk_private.go:119 +0x40
  sync.(*Once).doSlow()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/sync/once.go:74 +0xb4
  sync.(*Once).Do()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/sync/once.go:65 +0x40
  github.com/projectdiscovery/nuclei/v3/lib.(*NucleiEngine).init()
      /Users/tarun/Codebase2/nuclei/lib/sdk_private.go:118 +0x2a8
  github.com/projectdiscovery/nuclei/v3/lib.NewThreadSafeNucleiEngineCtx()
      /Users/tarun/Codebase2/nuclei/lib/multi.go:97 +0x2a0
  github.com/projectdiscovery/nuclei/v3/lib/tests_test.TestThreadSafeNuclei.func1()
      /Users/tarun/Codebase2/nuclei/lib/tests/sdk_test.go:106 +0x9c
  github.com/projectdiscovery/nuclei/v3/lib/tests_test.TestThreadSafeNuclei()
      /Users/tarun/Codebase2/nuclei/lib/tests/sdk_test.go:133 +0x270
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/testing/testing.go:1689 +0x180
  testing.(*T).Run.gowrap1()
      /opt/homebrew/Cellar/go/1.22.6/libexec/src/testing/testing.go:1742 +0x40

Previous read at 0x000106f1b918 by goroutine 9706:
  github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate.StartActiveMemGuardian.func1()
      /Users/tarun/Codebase2/nuclei/pkg/protocols/common/protocolstate/memguardian.go:33 +0x50

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[dwisiswant0]

q & todos:

• is FQDN == raw input (DNS-protocol-based template)?
• configurable git HTTP client