fix missing browser init

[Mzack9999]

Proposed changes

Closes #5880

Example

$ cat example.go
package main

import (
	"context"
	"sync"

	nuclei "github.com/projectdiscovery/nuclei/v3/lib"
)

func main() {

	ctx := context.Background()
	ne, err := nuclei.NewThreadSafeNucleiEngineCtx(ctx,
		// nuclei.WithVerbosity(nuclei.VerbosityOptions{
		// 	Verbose: true,
		// 	Debug:   true,
		// }),
		nuclei.EnableHeadlessWithOpts(&nuclei.HeadlessOpts{
			PageTimeout: 30,
			ShowBrowser: false,
			UseChrome:   false,
		}),
	)
	if err != nil {
		panic(err)
	}
	// setup waitgroup to handle concurrency
	wg := &sync.WaitGroup{}

	// scan 1 = run dns templates on scanme.sh
	wg.Add(1)
	go func() {
		defer wg.Done()
		err = ne.ExecuteNucleiWithOpts([]string{"scanme.sh"}, nuclei.WithTemplateFilters(nuclei.TemplateFilters{IDs: []string{"screenshot"}}))
		if err != nil {
			panic(err)
		}
	}()

	// scan 2 = run http templates on honey.scanme.sh
	wg.Add(1)
	go func() {
		defer wg.Done()
		err = ne.ExecuteNucleiWithOpts([]string{"honey.scanme.sh"}, nuclei.WithTemplateFilters(nuclei.TemplateFilters{IDs: []string{"screenshot"}}))
		if err != nil {
			panic(err)
		}
	}()

	// wait for all scans to finish
	wg.Wait()
	defer ne.Close()
}
$ go run .
$

Summary by CodeRabbit

• New Features

  • Added a Browser field to enhance execution options within the Nuclei engine.
  • Improved request handling capabilities by integrating a customizable HTTP client.

• Bug Fixes

  • Updated error formatting in output for improved clarity in error messages during tests.
  • Adjusted template validation handling to bypass strict checks temporarily.

• Chores

  • Updated dependency status and handled version retraction for improved project stability.
  • Modified multi-protocol request test case configuration for refined HTTP matching criteria.

[coderabbitai]

/

[jeyaseelans86]

Getting new errors now

While running with https:// or http:// protocol

err = ne.ExecuteNucleiWithOpts([]string{"https://scanme.sh"}, nuclei.WithTemplateFilters(nuclei.TemplateFilters{IDs: []string{"screenshot"}}))

Gives me

[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1057c9734]

goroutine 9766 [running]:
github.com/projectdiscovery/fastdialer/fastdialer.(*Dialer).DialTLS(0x0, {0x107d6de70, 0x140026ee0f0}, {0x106ba6dcb, 0x3}, {0x140083f7181, 0xd})
        /Users/jeya-12107/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.2.10/fastdialer/dialer.go:212 +0x34
github.com/projectdiscovery/nuclei/v3/pkg/protocols/headless/engine.newHttpClient.func1({0x107d6de70?, 0x140026ee0f0?}, {0x106ba6dcb?, 0x1052faa8c?}, {0x140083f7181?, 0x140078f4f00?})
        /Users/jeya-12107/go/pkg/mod/github.com/projectdiscovery/nuclei/v3@v3.3.7/pkg/protocols/headless/engine/http_client.go:52 +0xa8
net/http.(*Transport).customDialTLS(0x140092eea78?, {0x107d6de70?, 0x140026ee0f0?}, {0x106ba6dcb?, 0x30bc93e806994e58?}, {0x140083f7181?, 0x140092eea88?})
        /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/transport.go:1418 +0x50
net/http.(*Transport).dialConn(0x14000448300, {0x107d6de70, 0x140026ee0f0}, {{}, 0x0, {0x14006994e58, 0x5}, {0x140083f7181, 0xd}, 0x0})
        /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/transport.go:1704 +0x2f0
net/http.(*Transport).dialConnFor(0x14000448300, 0x1400287a9a0)
        /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/transport.go:1563 +0x8c
net/http.(*Transport).startDialConnForLocked.func1()
        /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/transport.go:1545 +0x34
created by net/http.(*Transport).startDialConnForLocked in goroutine 9765
        /opt/homebrew/Cellar/go/1.23.1/libexec/src/net/http/transport.go:1544 +0x118
exit status 2 ``` 

While running without https:// or http:// protocol , Getting this error

err = ne.ExecuteNucleiWithOpts([]string{"scanme.sh"}, nuclei.WithTemplateFilters(nuclei.TemplateFilters{IDs: []string{"screenshot"}}))

[WRN] [screenshot] Could not execute request for scanme.sh: could not get html element: error occurred executing action: [:RUNTIME] could not navigate to url scanme.sh <- {-32000 Cannot navigate to invalid URL }

@Mzack9999 Pl check

[Mzack9999]

@jeyaseelans86 This should be fixed with latest commit, nice catch!

[dogancanbakir]

LGTM -some tests are failing

[Mzack9999]

@coderabbitai resolve

[Mzack9999]

@coderabbitai ignore

[coderabbitai]

✅ Actions performed

Comments resolved.

[coderabbitai]

✅ Actions performed

Reviews paused.

[Mzack9999]

@coderabbitai resolve

[Mzack9999]

@coderabbitai ignore

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

pkg/catalog/loader/loader.go (2)

Line range hint 271-273: Consider enhancing error handling

The error handling in the template loading process could be improved. Currently, errors are logged but the function continues processing, which could lead to incomplete template loading or silent failures.

Consider collecting errors during template loading and returning them as part of the function result:

-func (store *Store) LoadTemplatesOnlyMetadata() error {
+func (store *Store) LoadTemplatesOnlyMetadata() ([]error, error) {
+    var loadErrors []error
     templatePaths, errs := store.config.Catalog.GetTemplatesPath(store.finalTemplates)
     store.logErroredTemplates(errs)
+    if len(errs) > 0 {
+        loadErrors = append(loadErrors, errs...)
+    }

---

Line range hint 492-493: Consider adding worker pool for controlled concurrency

While the concurrent template loading implementation is good, it could benefit from a worker pool to control the number of concurrent goroutines, especially when dealing with a large number of templates.

Consider implementing a worker pool pattern:

+    const maxWorkers = 10
+    sem := make(chan struct{}, maxWorkers)
     for templatePath := range templatePathMap {
         wgLoadTemplates.Add(1)
+        sem <- struct{}{} // Acquire semaphore
         go func(templatePath string) {
-            defer wgLoadTemplates.Done()
+            defer func() {
+                <-sem // Release semaphore
+                wgLoadTemplates.Done()
+            }()

Also applies to: 513-514

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 02fdb79 and b26f43e.

📒 Files selected for processing (1)

• pkg/catalog/loader/loader.go (1 hunks)

🔇 Additional comments (1)

pkg/catalog/loader/loader.go (1)

396-398: ⚠️ Potential issue

Temporary fix needs tracking and timeline

The commented-out validation code is a temporary workaround for issue #11324. While this allows GitHub actions to run, disabling duplicate template ID validation could lead to potential issues:

1. Template ID collisions could cause unpredictable behavior
2. Security implications if malicious templates exploit the relaxed validation

Let's verify the current state of duplicate template IDs:

Follow-up actions needed:

1. Add a tracking issue for re-enabling validation
2. Set a timeline for the temporary bypass
3. Document the current state in release notes

✅ Verification successful

Based on the shell script results, I can see that there are indeed some duplicate template IDs in the test files, but these are intentional duplicates used for testing purposes (located under integration_tests/, pkg/templates/tests/, etc.). The duplicates are:

1. Test templates with ID "basic-example", "workflow-example", etc. in test directories
2. Test flow templates with ID "nuclei-flow-dns" in test directories
3. Other test-specific duplicates in integration test directories

Since all the duplicate IDs are in test files and not in actual templates, the temporary disabling of validation is a reasonable workaround that doesn't pose security risks in production.

Temporary fix is acceptable with current scope

The disabled validation only affects duplicate template ID checks in test files, which is the intended behavior. The change is properly documented with a TODO comment linking to issue #11324.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for duplicate template IDs in the codebase
# Expected results: List of template files with duplicate IDs to assess the impact

# Search for template ID declarations in yaml files
rg -l "id: " -g "*.yaml" | while read -r file; do
  id=$(awk '/^id:/ {print $2}' "$file")
  if [ ! -z "$id" ]; then
    echo "$id $file"
  fi
done | sort | uniq -d -w 36

Length of output: 49432