Streamline packaging #844

antonok-edm · 2024-02-09T00:09:59Z

The purpose of this PR is to trim the copy-pasted boilerplate at the end of each packaging script (generateCRXFile, util.installErrorHandlers, util.addCommonScriptOptions, keyDir, util.createTableIfNotExists, etc.).

Each package*.js script now contains a much more complete, declarative view of what goes into and comes out of each component. Slightly less-used functionality (e.g. --local-run, file content hashing) is now directly available for use by all of the scripts.

I took the added step of moving logic from generateAdBlockRustDataFiles.js and generateManifestForRustAdblock.js into packageAdBlock.js. I'd like to do this for other components, but I need to understand them better first.

This PR shrinks the codebase by about 300 lines. Note that most of the "added" lines consist of moved/reorganized code.

Scripts tested in staging:

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

scripts/packageTorPluggableTransports.js|57| [semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller
scripts/packageTorPluggableTransports.js|61| [semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller
scripts/packageWalletDataFiles.js|14| [semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller
scripts/packageWalletDataFiles.js|30| [semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller
scripts/packageWalletDataFiles.js|35| [semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:46Z

scripts/ntp-sponsored-images/package.js


-util.installErrorHandlers()
+    this.stagingDir = path.join(rootBuildDir, 'staging', this.platformRegion)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:46Z

scripts/ntp-sponsored-images/package.js


-util.installErrorHandlers()
+    this.stagingDir = path.join(rootBuildDir, 'staging', this.platformRegion)
+    this.crxFile = path.join(rootBuildDir, 'output', `ntp-sponsored-images-${this.platformRegion}.crx`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:46Z

scripts/ntp-sponsored-images/package.js

-  .parse(process.argv)
+  privateKeyFromDir (keyDir) {
+    // Desktop private key file names do not have the -desktop suffix, but android has -android
+    return path.join(keyDir, `ntp-sponsored-images-${this.platformRegion.replace('-desktop', '')}.pem`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:46Z

scripts/packageAdBlock.js

+  }
+
+  privateKeyFromDir (keyDir) {
+    return path.join(keyDir, `${this.crxName}.pem`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:46Z

scripts/packageAdBlock.js

+  async stageFiles (version, outputDir) {
+    const regions = await getRegionalLists()
+    const regionalCatalogString = JSON.stringify(regions)
+    const regionalCatalogPath = path.join(outputDir, 'regional_catalog.json')


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:48Z

scripts/packageNTPSuperReferrerComponent.js

+    this.componentId = componentId
+    this.superReferrerName = superReferrerName
+    this.stagingDir = path.join(rootBuildDir, 'staging', this.superReferrerName)
+    this.crxFile = path.join(rootBuildDir, 'output', `ntp-super-referrer-${this.superReferrerName}.crx`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:48Z

scripts/packageTorClient.js

-    .option('-d, --keys-directory <dir>', 'directory containing private keys for signing crx files', 'abc')
-    .option('-f, --key-file <file>', 'private key file for signing crx', 'key.pem'))
-  .parse(process.argv)
+    this.stagingDir = path.join('build', 'tor-client-updater', this.platform)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:48Z

scripts/packageTorClient.js

-    .option('-f, --key-file <file>', 'private key file for signing crx', 'key.pem'))
-  .parse(process.argv)
+    this.stagingDir = path.join('build', 'tor-client-updater', this.platform)
+    this.crxFile = path.join('build', 'tor-client-updater', `tor-client-updater-${this.platform}.crx`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:48Z

scripts/packageTorClient.js


-let keyParam = ''
+  privateKeyFromDir (keyDir) {
+    return path.join(keyDir, `tor-client-updater-${this.platform}.pem`)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

github-actions · 2024-02-09T00:11:48Z

scripts/packageTorPluggableTransports.js


-util.installErrorHandlers()
+    this.stagingDir = path.join('build', TOR_PLUGGABLE_TRANSPORTS_UPDATER, this.platform)


_{reported by reviewdog 🐶}
[semgrep] Detected possible user input going into a path.join or path.resolve function. This could possibly lead to a path traversal vulnerability, where the attacker can access arbitrary files stored in the file system. Instead, be sure to sanitize or validate user input first.

Source: https://semgrep.dev/r/javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal

Cc @thypon @bcaller

linhkikuchi · 2024-02-09T07:04:37Z

Nice, FYI I'm doing another PR here https://github.com/brave/devops/pull/11212 to shrink the codebase and avoid copy-paste for all extensions pipelines also

antonok-edm added 5 commits February 6, 2024 16:23

remove superfluous directory creation

f1c9cca

absorb more common packaging logic into util.prepareNextVersionCRX

83b48fd

use per-component classes to customize packaging behavior

6211b57

split packageEthereumRemoteClient and packageWalletDataFiles

f867cae

move common packaging logic into shared packageComponent script

263322d

antonok-edm requested review from fmarier, mihaiplesa and ShivanKaul February 9, 2024 00:09

antonok-edm self-assigned this Feb 9, 2024

github-actions bot reviewed Feb 9, 2024

View reviewed changes

github-actions bot added the needs-security-review label Feb 9, 2024

github-actions bot assigned bcaller and thypon Feb 9, 2024

antonok-edm mentioned this pull request Feb 9, 2024

Select which adblock data files to generate using cli args #699

Closed

mihaiplesa assigned linhkikuchi Feb 9, 2024

antonok-edm mentioned this pull request Feb 15, 2024

Restore the check for missing adblock manifests when packaging #848

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streamline packaging #844

Streamline packaging #844

antonok-edm commented Feb 9, 2024 •

edited

Loading

github-actions bot left a comment

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

linhkikuchi commented Feb 9, 2024


		util.installErrorHandlers()
		this.stagingDir = path.join(rootBuildDir, 'staging', this.platformRegion)


		util.installErrorHandlers()
		this.stagingDir = path.join('build', TOR_PLUGGABLE_TRANSPORTS_UPDATER, this.platform)

Streamline packaging #844

Are you sure you want to change the base?

Streamline packaging #844

Conversation

antonok-edm commented Feb 9, 2024 • edited Loading

Scripts tested in staging:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

linhkikuchi commented Feb 9, 2024

antonok-edm commented Feb 9, 2024 •

edited

Loading