feat: adding google malware detection #36

jsdevel · 2022-08-28T08:11:42Z

Test ipfs malware sample from google with lookup and evaluate.

vasco-santos

Thanks for the first iteration @jsdevel

Let's please create a package from the start as previously talked and specified in https://hackmd.io/Y1vEmm9qTBmQU9Wzc3PY2w. Once we hook up tests, KV to store results / KV Locker it will be a bunch of logic that should be isolated. Also, we will want the logs for metrics + trigger badbits mail

For the package, we want a worker quite similar to this one with:

wrangler.toml
index.js
cors.js
error-handler.js
env.js

packages/edge-gateway/src/gateway.js

vasco-santos

Thanks for the new iteration Joe. Added some initial feedback even though I know this is still WIP 😉

packages/cid-verifier/scripts/cli.js

packages/cid-verifier/scripts/heartbeat.js

packages/cid-verifier/package.json

packages/cid-verifier/src/bindings.d.ts

packages/cid-verifier/src/env.js

vasco-santos

Thanks for the new iteration. Glad things are shaping up. A couple of general comments I could not have a place to add in review:

github actions workflow for cid verifier package (can be based on edge-gateway
CI is having a bad time, looks like pnpm lockfile should be commited? perhaps run on root dir pnpm clean && pnpm i

Also, this is first set of comments and I did not get into the actual code for CID verification. Just all the boilerplate around

packages/cid-verifier/package.json

packages/cid-verifier/src/bindings.d.ts

packages/cid-verifier/package.json

packages/edge-gateway/wrangler.toml

packages/edge-gateway/src/utils/deny-list.js

packages/edge-gateway/package.json

vasco-santos

Extra thoughts, we should be good after this. Good job

packages/cid-verifier/wrangler.toml

packages/cid-verifier/test/utils/setup.js

packages/cid-verifier/src/verification.js

jsdevel · 2022-09-02T00:05:37Z

Thanks for the new iteration. Glad things are shaping up. A couple of general comments I could not have a place to add in review:

github actions workflow for cid verifier package (can be based on edge-gateway

CI is having a bad time, looks like pnpm lockfile should be commited? perhaps run on root dir pnpm clean && pnpm i

Also, this is first set of comments and I did not get into the actual code for CID verification. Just all the boilerplate around

i saw that error. i checked in the pnpm lock file. i think locally i'm using a later version of pnpm and that's causing it.

packages/cid-verifier/package.json

packages/cid-verifier/src/verification.js

jsdevel · 2022-09-08T22:38:35Z

New set of changes. Please also add README file, gather a copy from edge-gateway and basically need specific secrets for here specified there

i'll add a README once the approach is finalized

packages/cid-verifier/src/verification.js

jsdevel · 2022-09-08T20:53:25Z

packages/cid-verifier/src/bindings.d.ts

+export interface EnvInput {
+  ENV: string;
+  DEBUG: string;
+  GOOGLE_EVALUATE_SAFE_CONFIDENCE_LEVELS: Array<string>;


i had this in case there were other levels we'd want to configure other than SAFE

jsdevel · 2022-09-08T21:10:27Z

packages/cid-verifier/src/index.js

+  .get('/verification', withCorsHeaders(verificationGet))
+  .post('/verification', withCorsHeaders(verificationPost))


we can go over this in our call

packages/cid-verifier/src/verification.js

vasco-santos · 2022-09-09T08:43:53Z

packages/cid-verifier/src/bindings.d.ts

+export interface EnvInput {
+  ENV: string;
+  DEBUG: string;
+  GOOGLE_EVALUATE_SAFE_CONFIDENCE_LEVELS: Array<string>;


Ok, I got it thanks, this also needs to go to docs. And added in wrangler toml for other environments

packages/cid-verifier/src/verification.js

packages/edge-gateway/wrangler.toml

packages/cid-verifier/wrangler.toml

packages/cid-verifier/src/verification.js

* Moving DENYLIST logic to cid-verifier from edge-gateway

jsdevel

Hoping you can get this sample.html test to work.

jsdevel · 2022-09-10T02:49:59Z

packages/edge-gateway/test/index.spec.js

@@ -101,3 +101,10 @@ test('Gets content with other base encodings', async (t) => {
  await response.waitUntil()
  t.is(await response.text(), 'Hello dot.storage! 😎')
 })
+
+test('Sends HTML files to cid-verifier', async (t) => {


@vasco-santos i'm having trouble getting this to work. can you take a look?

vasco-santos

LGTM 🎉

jsdevel force-pushed the adding-google-malware-detection branch from 0fe2ded to 1b9c179 Compare August 28, 2022 08:13

jsdevel changed the title ~~Adding google malware detection.~~ feat: Adding google malware detection. Aug 28, 2022

jsdevel changed the title ~~feat: Adding google malware detection.~~ feat: adding google malware detection Aug 28, 2022

jsdevel force-pushed the adding-google-malware-detection branch from 1b9c179 to e6f4227 Compare August 28, 2022 08:14

vasco-santos requested changes Aug 29, 2022

View reviewed changes

packages/edge-gateway/src/gateway.js Outdated Show resolved Hide resolved

packages/edge-gateway/src/gateway.js Outdated Show resolved Hide resolved

packages/edge-gateway/src/gateway.js Outdated Show resolved Hide resolved

vasco-santos reviewed Aug 30, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from 9f4a736 to c3dd17c Compare September 1, 2022 09:13

vasco-santos previously requested changes Sep 1, 2022

View reviewed changes

vasco-santos reviewed Sep 1, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 16 times, most recently from de4735a to 57a0953 Compare September 7, 2022 07:01

vasco-santos reviewed Sep 7, 2022

View reviewed changes

packages/cid-verifier/package.json Outdated Show resolved Hide resolved

packages/cid-verifier/src/verification.js Outdated Show resolved Hide resolved

jsdevel force-pushed the adding-google-malware-detection branch from 57a0953 to cc9b4f3 Compare September 7, 2022 22:20

storacha deleted a comment from vasco-santos Sep 7, 2022

jsdevel force-pushed the adding-google-malware-detection branch from cf8bef9 to 5412487 Compare September 8, 2022 22:38

jsdevel commented Sep 8, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from 8833a3f to 3960f33 Compare September 9, 2022 04:05

jsdevel commented Sep 9, 2022

View reviewed changes

packages/cid-verifier/src/verification.js Show resolved Hide resolved

vasco-santos requested changes Sep 9, 2022

View reviewed changes

vasco-santos reviewed Sep 9, 2022

View reviewed changes

packages/cid-verifier/src/verification.js Outdated Show resolved Hide resolved

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from bd6d31c to 7d51262 Compare September 9, 2022 19:15

feat: adding google malware detection

65a107e

* Moving DENYLIST logic to cid-verifier from edge-gateway

jsdevel force-pushed the adding-google-malware-detection branch from 7d51262 to 65a107e Compare September 10, 2022 02:48

jsdevel commented Sep 10, 2022

View reviewed changes

test: requests to cid verifier with html response trigger verification

f3c6b84

vasco-santos force-pushed the adding-google-malware-detection branch from 6114aa7 to f3c6b84 Compare September 12, 2022 10:02

vasco-santos added 2 commits September 12, 2022 12:07

chore: add retries to kv

92978b8

chore: remove hearbeat

27c095f

vasco-santos approved these changes Sep 12, 2022

View reviewed changes

vasco-santos merged commit bacaeae into main Sep 12, 2022

vasco-santos deleted the adding-google-malware-detection branch September 12, 2022 10:25

This was referenced Sep 12, 2022

chore(main): release edge-gateway 1.2.0 #54

Merged

chore(main): release cid-verifier 1.0.0 #55

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adding google malware detection #36

feat: adding google malware detection #36

jsdevel commented Aug 28, 2022 •

edited

Loading

vasco-santos left a comment •

edited

Loading

vasco-santos left a comment

vasco-santos left a comment

vasco-santos left a comment

jsdevel commented Sep 2, 2022

jsdevel commented Sep 8, 2022

jsdevel Sep 8, 2022

jsdevel Sep 8, 2022

vasco-santos Sep 9, 2022 •

edited

Loading

jsdevel left a comment

jsdevel Sep 10, 2022

vasco-santos left a comment

		.get('/verification', withCorsHeaders(verificationGet))
		.post('/verification', withCorsHeaders(verificationPost))

feat: adding google malware detection #36

feat: adding google malware detection #36

Conversation

jsdevel commented Aug 28, 2022 • edited Loading

vasco-santos left a comment • edited Loading

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

jsdevel commented Sep 2, 2022

jsdevel commented Sep 8, 2022

jsdevel Sep 8, 2022

Choose a reason for hiding this comment

jsdevel Sep 8, 2022

Choose a reason for hiding this comment

vasco-santos Sep 9, 2022 • edited Loading

Choose a reason for hiding this comment

jsdevel left a comment

Choose a reason for hiding this comment

jsdevel Sep 10, 2022

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

jsdevel commented Aug 28, 2022 •

edited

Loading

vasco-santos left a comment •

edited

Loading

vasco-santos Sep 9, 2022 •

edited

Loading