Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch JSONL Output #5705

Merged
merged 2 commits into from
Nov 15, 2024
Merged

Batch JSONL Output #5705

merged 2 commits into from
Nov 15, 2024

Conversation

kchason
Copy link
Contributor

@kchason kchason commented Oct 8, 2024

Proposed changes

Similar to the batch logic in #5688, this adds the ability to batch outputs in the JSONL export to avoid retaining the entire result set in memory.

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@auto-assign auto-assign bot requested a review from dogancanbakir October 8, 2024 12:10
@GeorginaReeder
Copy link

Thanks so much for your contribution @kchason ! :)

@ehsandeep ehsandeep mentioned this pull request Nov 14, 2024
Open
@ehsandeep ehsandeep merged commit ba6a050 into projectdiscovery:dev Nov 15, 2024
11 of 14 checks passed
@BrewTestBot BrewTestBot mentioned this pull request Nov 21, 2024
Merged
@blsaccess blsaccess mentioned this pull request Nov 22, 2024
Merged
Explorer1092 pushed a commit to Explorer1092/nuclei that referenced this pull request Nov 25, 2024
Merge branch 'projectdiscovery-main'
0344a14 
* projectdiscovery-main: (283 commits)
  upgrade
  ci(generate-docs): push w/o pull (projectdiscovery#5843)
  auto gen docs update
  handle env variables in dynamic secret file (projectdiscovery#5835)
  disable self-contained and file protocol templates as default (projectdiscovery#5825)
  chore(deps): bump github.com/projectdiscovery/hmap from 0.0.65 to 0.0.67 (projectdiscovery#5832)
  feat: Added time based delay analyzer to fuzzing implementation (projectdiscovery#5781)
  fix: data race at `protocolstate`, `contextargs` & some outdated test cases (projectdiscovery#5820)
  chore(deps): bump github.com/projectdiscovery/retryabledns
  chore(deps): bump github.com/projectdiscovery/goflags
  chore(deps): bump github.com/projectdiscovery/dsl from 0.3.0 to 0.3.3
  chore(deps): bump github.com/projectdiscovery/rawhttp
  Batch JSONL output and add trailing commas (projectdiscovery#5705)
  ci: refactor workflows (projectdiscovery#5818)
  chore(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (projectdiscovery#5795)
  chore(deps): bump github.com/projectdiscovery/rawhttp (projectdiscovery#5809)
  chore(deps): bump github.com/projectdiscovery/wappalyzergo (projectdiscovery#5808)
  update version
  chore(deps): bump github.com/projectdiscovery/gologger
  chore(deps): bump github.com/projectdiscovery/useragent
  ...

# Conflicts:
#	.github/ISSUE_TEMPLATE/feature_request.md
#	.github/ISSUE_TEMPLATE/issue-report.md
#	.run/DSLFunctionsIT.run.xml
#	DESIGN.md
#	README.md
#	README_CN.md
#	README_ID.md
#	README_KR.md
#	cmd/docgen/docgen.go
#	cmd/functional-test/main.go
#	cmd/functional-test/run.sh
#	cmd/integration-test/custom-dir.go
#	cmd/integration-test/dns.go
#	cmd/integration-test/file.go
#	cmd/integration-test/fuzz.go
#	cmd/integration-test/headless.go
#	cmd/integration-test/http.go
#	cmd/integration-test/integration-test.go
#	cmd/integration-test/library.go
#	cmd/integration-test/loader.go
#	cmd/integration-test/offline-http.go
#	cmd/integration-test/ssl.go
#	cmd/integration-test/template-dir.go
#	cmd/integration-test/template-path.go
#	cmd/integration-test/websocket.go
#	cmd/integration-test/whois.go
#	cmd/integration-test/workflow.go
#	cmd/nuclei/main.go
#	examples/advanced/advanced.go
#	examples/simple/simple.go
#	go.mod
#	go.sum
#	internal/colorizer/colorizer.go
#	internal/runner/banner.go
#	internal/runner/healthcheck.go
#	internal/runner/lazy.go
#	internal/runner/options.go
#	internal/runner/proxy.go
#	internal/runner/runner.go
#	internal/runner/runner_test.go
#	internal/runner/templates.go
#	lib/example_test.go
#	lib/sdk.go
#	lib/sdk_private.go
#	pkg/catalog/disk/find.go
#	pkg/catalog/loader/filter/path_filter.go
#	pkg/catalog/loader/loader.go
#	pkg/catalog/loader/loader_test.go
#	pkg/catalog/loader/remote_loader.go
#	pkg/core/engine.go
#	pkg/core/executors.go
#	pkg/core/workflow_execute.go
#	pkg/core/workflow_execute_test.go
#	pkg/external/customtemplates/azure_blob.go
#	pkg/external/customtemplates/github.go
#	pkg/external/customtemplates/github_test.go
#	pkg/external/customtemplates/gitlab.go
#	pkg/external/customtemplates/s3.go
#	pkg/external/customtemplates/templates_provider.go
#	pkg/fuzz/component/path.go
#	pkg/fuzz/fuzz.go
#	pkg/input/formats/swagger/swagger.go
#	pkg/input/provider/list/hmap.go
#	pkg/input/provider/list/hmap_test.go
#	pkg/input/transform.go
#	pkg/installer/template.go
#	pkg/installer/template_test.go
#	pkg/installer/util.go
#	pkg/installer/versioncheck.go
#	pkg/installer/versioncheck_test.go
#	pkg/js/compiler/compiler.go
#	pkg/js/compiler/init.go
#	pkg/js/global/scripts.go
#	pkg/js/libs/mssql/mssql.go
#	pkg/js/libs/postgres/postgres.go
#	pkg/loader/workflow/workflow_loader.go
#	pkg/model/model.go
#	pkg/model/model_test.go
#	pkg/model/types/severity/severities.go
#	pkg/model/types/stringslice/stringslice.go
#	pkg/operators/common/dsl/dsl.go
#	pkg/operators/extractors/compile.go
#	pkg/operators/extractors/extract.go
#	pkg/operators/matchers/compile.go
#	pkg/operators/matchers/match.go
#	pkg/operators/operators.go
#	pkg/output/format_screen.go
#	pkg/output/output.go
#	pkg/output/output_test.go
#	pkg/protocols/common/expressions/expressions.go
#	pkg/protocols/common/expressions/variables.go
#	pkg/protocols/common/generators/generators.go
#	pkg/protocols/common/generators/generators_test.go
#	pkg/protocols/common/generators/options.go
#	pkg/protocols/common/generators/validate.go
#	pkg/protocols/common/helpers/eventcreator/eventcreator.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
#	pkg/protocols/common/helpers/writer/writer.go
#	pkg/protocols/common/interactsh/interactsh.go
#	pkg/protocols/common/interactsh/options.go
#	pkg/protocols/common/protocolstate/state.go
#	pkg/protocols/common/replacer/replacer.go
#	pkg/protocols/common/uncover/uncover.go
#	pkg/protocols/common/utils/vardump/dump.go
#	pkg/protocols/common/variables/variables.go
#	pkg/protocols/dns/dns.go
#	pkg/protocols/dns/dns_test.go
#	pkg/protocols/dns/dnsclientpool/clientpool.go
#	pkg/protocols/dns/operators.go
#	pkg/protocols/dns/operators_test.go
#	pkg/protocols/dns/request.go
#	pkg/protocols/dns/request_test.go
#	pkg/protocols/file/file.go
#	pkg/protocols/file/find_test.go
#	pkg/protocols/file/operators.go
#	pkg/protocols/file/operators_test.go
#	pkg/protocols/file/request.go
#	pkg/protocols/file/request_test.go
#	pkg/protocols/headless/engine/engine.go
#	pkg/protocols/headless/engine/http_client.go
#	pkg/protocols/headless/engine/instance.go
#	pkg/protocols/headless/engine/page_actions.go
#	pkg/protocols/headless/engine/page_actions_test.go
#	pkg/protocols/headless/engine/util.go
#	pkg/protocols/headless/headless.go
#	pkg/protocols/headless/operators.go
#	pkg/protocols/headless/request.go
#	pkg/protocols/http/build_request.go
#	pkg/protocols/http/build_request_test.go
#	pkg/protocols/http/cluster.go
#	pkg/protocols/http/http.go
#	pkg/protocols/http/http_test.go
#	pkg/protocols/http/httpclientpool/clientpool.go
#	pkg/protocols/http/operators.go
#	pkg/protocols/http/operators_test.go
#	pkg/protocols/http/request.go
#	pkg/protocols/http/request_annotations.go
#	pkg/protocols/http/request_annotations_test.go
#	pkg/protocols/http/request_fuzz.go
#	pkg/protocols/http/request_generator.go
#	pkg/protocols/http/request_generator_test.go
#	pkg/protocols/http/signature.go
#	pkg/protocols/http/signer/signer.go
#	pkg/protocols/http/signerpool/signerpool.go
#	pkg/protocols/javascript/js.go
#	pkg/protocols/network/network.go
#	pkg/protocols/network/network_test.go
#	pkg/protocols/network/networkclientpool/clientpool.go
#	pkg/protocols/network/operators.go
#	pkg/protocols/network/operators_test.go
#	pkg/protocols/network/request.go
#	pkg/protocols/network/request_test.go
#	pkg/protocols/offlinehttp/find_test.go
#	pkg/protocols/offlinehttp/offlinehttp.go
#	pkg/protocols/offlinehttp/operators.go
#	pkg/protocols/offlinehttp/operators_test.go
#	pkg/protocols/offlinehttp/request.go
#	pkg/protocols/protocols.go
#	pkg/protocols/ssl/ssl.go
#	pkg/protocols/ssl/ssl_test.go
#	pkg/protocols/utils/http/variables.go
#	pkg/protocols/utils/http/variables_test.go
#	pkg/protocols/utils/utils.go
#	pkg/protocols/websocket/websocket.go
#	pkg/protocols/whois/rdapclientpool/clientpool.go
#	pkg/protocols/whois/whois.go
#	pkg/reporting/client.go
#	pkg/reporting/dedupe/dedupe.go
#	pkg/reporting/dedupe/dedupe_test.go
#	pkg/reporting/exporters/es/elasticsearch.go
#	pkg/reporting/exporters/jsonexporter/jsonexporter.go
#	pkg/reporting/exporters/markdown/markdown.go
#	pkg/reporting/exporters/sarif/sarif.go
#	pkg/reporting/exporters/splunk/splunkhec.go
#	pkg/reporting/format/format_utils.go
#	pkg/reporting/format/format_utils_test.go
#	pkg/reporting/options.go
#	pkg/reporting/reporting.go
#	pkg/reporting/trackers/jira/jira.go
#	pkg/scan/scan_context.go
#	pkg/templates/cluster.go
#	pkg/templates/compile.go
#	pkg/templates/compile_test.go
#	pkg/templates/parser_test.go
#	pkg/templates/tag_filter.go
#	pkg/templates/tag_filter_test.go
#	pkg/templates/templates.go
#	pkg/templates/templates_doc_examples.go
#	pkg/templates/types/types.go
#	pkg/templates/workflows.go
#	pkg/testutils/testutils.go
#	pkg/tmplexec/exec.go
#	pkg/tmplexec/flow/flow_executor.go
#	pkg/tmplexec/interface.go
#	pkg/tmplexec/multiproto/multi.go
#	pkg/tmplexec/multiproto/multi_test.go
#	pkg/types/interfaces.go
#	pkg/types/resume.go
#	pkg/types/types.go
#	pkg/utils/template_path.go
#	pkg/utils/utils.go
#	pkg/utils/yaml/preprocess.go
#	pkg/workflows/workflows.go
#	pkg/workflows/workflows_test.go
Explorer1092 pushed a commit to Explorer1092/nuclei that referenced this pull request Nov 25, 2024
Merge branch 'projectdiscovery-main'
74103df 
* projectdiscovery-main: (283 commits)
  upgrade
  ci(generate-docs): push w/o pull (projectdiscovery#5843)
  auto gen docs update
  handle env variables in dynamic secret file (projectdiscovery#5835)
  disable self-contained and file protocol templates as default (projectdiscovery#5825)
  chore(deps): bump github.com/projectdiscovery/hmap from 0.0.65 to 0.0.67 (projectdiscovery#5832)
  feat: Added time based delay analyzer to fuzzing implementation (projectdiscovery#5781)
  fix: data race at `protocolstate`, `contextargs` & some outdated test cases (projectdiscovery#5820)
  chore(deps): bump github.com/projectdiscovery/retryabledns
  chore(deps): bump github.com/projectdiscovery/goflags
  chore(deps): bump github.com/projectdiscovery/dsl from 0.3.0 to 0.3.3
  chore(deps): bump github.com/projectdiscovery/rawhttp
  Batch JSONL output and add trailing commas (projectdiscovery#5705)
  ci: refactor workflows (projectdiscovery#5818)
  chore(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (projectdiscovery#5795)
  chore(deps): bump github.com/projectdiscovery/rawhttp (projectdiscovery#5809)
  chore(deps): bump github.com/projectdiscovery/wappalyzergo (projectdiscovery#5808)
  update version
  chore(deps): bump github.com/projectdiscovery/gologger
  chore(deps): bump github.com/projectdiscovery/useragent
  ...

# Conflicts:
#	.github/ISSUE_TEMPLATE/feature_request.md
#	.github/ISSUE_TEMPLATE/issue-report.md
#	.run/DSLFunctionsIT.run.xml
#	DESIGN.md
#	README.md
#	README_CN.md
#	README_ID.md
#	README_KR.md
#	cmd/docgen/docgen.go
#	cmd/functional-test/main.go
#	cmd/functional-test/run.sh
#	cmd/integration-test/custom-dir.go
#	cmd/integration-test/dns.go
#	cmd/integration-test/file.go
#	cmd/integration-test/fuzz.go
#	cmd/integration-test/headless.go
#	cmd/integration-test/http.go
#	cmd/integration-test/integration-test.go
#	cmd/integration-test/library.go
#	cmd/integration-test/loader.go
#	cmd/integration-test/offline-http.go
#	cmd/integration-test/ssl.go
#	cmd/integration-test/template-dir.go
#	cmd/integration-test/template-path.go
#	cmd/integration-test/websocket.go
#	cmd/integration-test/whois.go
#	cmd/integration-test/workflow.go
#	cmd/nuclei/main.go
#	examples/advanced/advanced.go
#	examples/simple/simple.go
#	go.mod
#	go.sum
#	internal/colorizer/colorizer.go
#	internal/runner/banner.go
#	internal/runner/healthcheck.go
#	internal/runner/lazy.go
#	internal/runner/options.go
#	internal/runner/proxy.go
#	internal/runner/runner.go
#	internal/runner/runner_test.go
#	internal/runner/templates.go
#	lib/example_test.go
#	lib/sdk.go
#	lib/sdk_private.go
#	pkg/catalog/disk/find.go
#	pkg/catalog/loader/filter/path_filter.go
#	pkg/catalog/loader/loader.go
#	pkg/catalog/loader/loader_test.go
#	pkg/catalog/loader/remote_loader.go
#	pkg/core/engine.go
#	pkg/core/executors.go
#	pkg/core/workflow_execute.go
#	pkg/core/workflow_execute_test.go
#	pkg/external/customtemplates/azure_blob.go
#	pkg/external/customtemplates/github.go
#	pkg/external/customtemplates/github_test.go
#	pkg/external/customtemplates/gitlab.go
#	pkg/external/customtemplates/s3.go
#	pkg/external/customtemplates/templates_provider.go
#	pkg/fuzz/component/path.go
#	pkg/fuzz/fuzz.go
#	pkg/input/formats/swagger/swagger.go
#	pkg/input/provider/list/hmap.go
#	pkg/input/provider/list/hmap_test.go
#	pkg/input/transform.go
#	pkg/installer/template.go
#	pkg/installer/template_test.go
#	pkg/installer/util.go
#	pkg/installer/versioncheck.go
#	pkg/installer/versioncheck_test.go
#	pkg/js/compiler/compiler.go
#	pkg/js/compiler/init.go
#	pkg/js/global/scripts.go
#	pkg/js/libs/mssql/mssql.go
#	pkg/js/libs/postgres/postgres.go
#	pkg/loader/workflow/workflow_loader.go
#	pkg/model/model.go
#	pkg/model/model_test.go
#	pkg/model/types/severity/severities.go
#	pkg/model/types/stringslice/stringslice.go
#	pkg/operators/common/dsl/dsl.go
#	pkg/operators/extractors/compile.go
#	pkg/operators/extractors/extract.go
#	pkg/operators/matchers/compile.go
#	pkg/operators/matchers/match.go
#	pkg/operators/operators.go
#	pkg/output/format_screen.go
#	pkg/output/output.go
#	pkg/output/output_test.go
#	pkg/protocols/common/expressions/expressions.go
#	pkg/protocols/common/expressions/variables.go
#	pkg/protocols/common/generators/generators.go
#	pkg/protocols/common/generators/generators_test.go
#	pkg/protocols/common/generators/options.go
#	pkg/protocols/common/generators/validate.go
#	pkg/protocols/common/helpers/eventcreator/eventcreator.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
#	pkg/protocols/common/helpers/writer/writer.go
#	pkg/protocols/common/interactsh/interactsh.go
#	pkg/protocols/common/interactsh/options.go
#	pkg/protocols/common/protocolstate/state.go
#	pkg/protocols/common/replacer/replacer.go
#	pkg/protocols/common/uncover/uncover.go
#	pkg/protocols/common/utils/vardump/dump.go
#	pkg/protocols/common/variables/variables.go
#	pkg/protocols/dns/dns.go
#	pkg/protocols/dns/dns_test.go
#	pkg/protocols/dns/dnsclientpool/clientpool.go
#	pkg/protocols/dns/operators.go
#	pkg/protocols/dns/operators_test.go
#	pkg/protocols/dns/request.go
#	pkg/protocols/dns/request_test.go
#	pkg/protocols/file/file.go
#	pkg/protocols/file/find_test.go
#	pkg/protocols/file/operators.go
#	pkg/protocols/file/operators_test.go
#	pkg/protocols/file/request.go
#	pkg/protocols/file/request_test.go
#	pkg/protocols/headless/engine/engine.go
#	pkg/protocols/headless/engine/http_client.go
#	pkg/protocols/headless/engine/instance.go
#	pkg/protocols/headless/engine/page_actions.go
#	pkg/protocols/headless/engine/page_actions_test.go
#	pkg/protocols/headless/engine/util.go
#	pkg/protocols/headless/headless.go
#	pkg/protocols/headless/operators.go
#	pkg/protocols/headless/request.go
#	pkg/protocols/http/build_request.go
#	pkg/protocols/http/build_request_test.go
#	pkg/protocols/http/cluster.go
#	pkg/protocols/http/http.go
#	pkg/protocols/http/http_test.go
#	pkg/protocols/http/httpclientpool/clientpool.go
#	pkg/protocols/http/operators.go
#	pkg/protocols/http/operators_test.go
#	pkg/protocols/http/request.go
#	pkg/protocols/http/request_annotations.go
#	pkg/protocols/http/request_annotations_test.go
#	pkg/protocols/http/request_fuzz.go
#	pkg/protocols/http/request_generator.go
#	pkg/protocols/http/request_generator_test.go
#	pkg/protocols/http/signature.go
#	pkg/protocols/http/signer/signer.go
#	pkg/protocols/http/signerpool/signerpool.go
#	pkg/protocols/javascript/js.go
#	pkg/protocols/network/network.go
#	pkg/protocols/network/network_test.go
#	pkg/protocols/network/networkclientpool/clientpool.go
#	pkg/protocols/network/operators.go
#	pkg/protocols/network/operators_test.go
#	pkg/protocols/network/request.go
#	pkg/protocols/network/request_test.go
#	pkg/protocols/offlinehttp/find_test.go
#	pkg/protocols/offlinehttp/offlinehttp.go
#	pkg/protocols/offlinehttp/operators.go
#	pkg/protocols/offlinehttp/operators_test.go
#	pkg/protocols/offlinehttp/request.go
#	pkg/protocols/protocols.go
#	pkg/protocols/ssl/ssl.go
#	pkg/protocols/ssl/ssl_test.go
#	pkg/protocols/utils/http/variables.go
#	pkg/protocols/utils/http/variables_test.go
#	pkg/protocols/utils/utils.go
#	pkg/protocols/websocket/websocket.go
#	pkg/protocols/whois/rdapclientpool/clientpool.go
#	pkg/protocols/whois/whois.go
#	pkg/reporting/client.go
#	pkg/reporting/dedupe/dedupe.go
#	pkg/reporting/dedupe/dedupe_test.go
#	pkg/reporting/exporters/es/elasticsearch.go
#	pkg/reporting/exporters/jsonexporter/jsonexporter.go
#	pkg/reporting/exporters/markdown/markdown.go
#	pkg/reporting/exporters/sarif/sarif.go
#	pkg/reporting/exporters/splunk/splunkhec.go
#	pkg/reporting/format/format_utils.go
#	pkg/reporting/format/format_utils_test.go
#	pkg/reporting/options.go
#	pkg/reporting/reporting.go
#	pkg/reporting/trackers/jira/jira.go
#	pkg/scan/scan_context.go
#	pkg/templates/cluster.go
#	pkg/templates/compile.go
#	pkg/templates/compile_test.go
#	pkg/templates/parser_test.go
#	pkg/templates/tag_filter.go
#	pkg/templates/tag_filter_test.go
#	pkg/templates/templates.go
#	pkg/templates/templates_doc_examples.go
#	pkg/templates/types/types.go
#	pkg/templates/workflows.go
#	pkg/testutils/testutils.go
#	pkg/tmplexec/exec.go
#	pkg/tmplexec/flow/flow_executor.go
#	pkg/tmplexec/interface.go
#	pkg/tmplexec/multiproto/multi.go
#	pkg/tmplexec/multiproto/multi_test.go
#	pkg/types/interfaces.go
#	pkg/types/resume.go
#	pkg/types/types.go
#	pkg/utils/template_path.go
#	pkg/utils/utils.go
#	pkg/utils/yaml/preprocess.go
#	pkg/workflows/workflows.go
#	pkg/workflows/workflows_test.go
@bf-rbrown bf-rbrown mentioned this pull request Nov 27, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dogancanbakir dogancanbakir dogancanbakir approved these changes

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kchason @GeorginaReeder @ehsandeep @dogancanbakir