Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: send email notifications for storage quota usage #1273

Merged
merged 30 commits into from
May 25, 2022

Conversation

adamalton
Copy link
Contributor

@adamalton adamalton commented Apr 25, 2022

Fixes #851.

This is an overall feature PR for #851, bringing together the work done in these sub-tasks:

Now that those sub-tasks are done and merged to this branch, this is ready to be reviewed and merged as a complete feature. Note that potential follow-up improvements for this feature are listed in #1283.

Some of the requirements in the #851, are already in main or are part of admin.

Specifically:

  • Manual interface to set a restriction flag (in admin.storage)
  • Set a new user tag with storage in admin.storage
  • Implement restriction on all APIs/interface

Outstanding discussion

There's an outstanding question of how often we want to run the cron job. Running it more frequently won't send the emails more frequently (each email type has its own frequency limit, so it will only be sent as often as necessary), but if we only run the cron every 24 hours then there's potentially up to a 24 hour delay between a user hitting a threshold and them receiving an email about it. If that delay is too long then we should run the cron more frequently. That needs to be balanced with not hammering the platform/DB too much (although I don't foresee a real issue here). My proposal is to increase the frequency to every 6 hours.
Original comment here

Oustanding tasks

  • We need the API key from the Mailchimp account adding to the GitHub secrets as MAILCHIMP_API_KEY. I don't think any of us (Potato) have got access to the right stuff in GitHub to be able to do this.
  • Testing this on staging before we deploy to production. (See below.)
  • Enabling the cron job for production as well as staging in the GitHub workflow config file and update cron schedule to run every 6 hours.

Testing this

Before we release this to prod, we should test it on staging. The plan for this is:

  1. Deploy to staging. (We can actually deploy to prod as well if we want, as the cron job config is only set up to run it on staging at the moment - see above.)
  2. Create some user accounts for ourselves and upload/pin some files.
  3. Use admin.storage to set our quotas to something low enough to reach the 75%/80%/etc limit.
  4. Run the cron job manually via the GitHub UI (see instructions).
  5. Reduce our quotas again so that we go over the limit.
  6. Manually trigger the cron again.
  7. Check that we’ve received the right emails and that the list of users over their quota has been sent to support@web3.storage.

@adamalton adamalton marked this pull request as ready for review May 17, 2022 14:54
Copy link
Contributor

@hugomrdias hugomrdias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just left one inline comment

im good with cron running every 6 hours.

Are we just missing an Mailchimp key to test this in staging ?

// If there's an overall problem with the request, the response is an object, but if it's
// successful or if there's a problem with the message-level part of the request, it gives an
// array of responses, one for each recipient.
let response = await this.mailchimpTx.messages.sendTemplate(messageDef)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we just use fetch here to call the api instead of requiring the full SDK, it seems to be just this code https://github.com/mailchimp/mailchimp-transactional-node/blob/master/src/ApiClient.js#L70-L104 running

…ction

Renaming the more recent migration to ensure we run them in the correct order.
Combing the two sets of changes together outside of the merge commit, so that it's clear to see what's changed.
@flea89 flea89 merged commit 0b1eb09 into main May 25, 2022
@flea89 flea89 deleted the feat/storage-quota-emails branch May 25, 2022 15:08
orvn added a commit that referenced this pull request Jun 13, 2022
* fix: JS errors from docs changes (#1334)

* fix: remove React fragment which is causing an error

This was causing `Each child in a list should have a unique "key" prop`

* fix: indentation from my previous commit

* fix: js errors

* fix: revert package-lock changes

* fix: lint error

Co-authored-by: Adam Alton <adamalton@gmail.com>

* fix: Removed mistakenly generated link on CID header item in filemanager (#1336)

* fix!: psa pinning APIs - rename requestId to requestid

* chore(main): release website 2.4.0 (#1299)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* make tooltip accessible (#1340)

* chore: rename pinned to psaPinned (#1268)


Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

* docs: update peers (#1344)

* feat: respond with unique error message when blocked API key is used (#1302)

* chore(main): release api 6.0.0 (#1325)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* chore(main): release website 2.4.1 (#1342)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix: Set fetch date before changing isFetching state (#1341)

* feat: send email notifications for storage quota usage (#1273)

This includes:
- a general email component in cron package for sending emails
- notifications to web3.storage users when they get to specific thresholds
- notifications to web3.storage admins when users go over their quota


Co-authored-by: Gary Homewood <gary@potatolondon.com>
Co-authored-by: Paolo <paolo@potatolondon.com>
Co-authored-by: Oli Evans <oli@tableflip.io>
Co-authored-by: francois-potato <103063611+francois-potato@users.noreply.github.com>

* fix: inaccurate used_storage migrations (#1360)

* feat: add user blocking functionality to web3 (#1322)

* chore: do not convert bigint to number (#1366)

* chore: use mailchimp provider in crons (#1368)

* chore: send list of storage quota violators to support@web3.storage, not admin@ (#1369)

* chore: remove unnecessary migration for creating admin user (#1373)

Rename the subsequent migration to keep the numbers sequential.
Now that we're using support@ rather than admin@ for the admin email address, that user already exists on both staging and prod.

* feat: implement postgres optimization (#1305)

* Refactor backups
* Add db migration scripts
* Update db configuration

* fix: typo in Logging constructor (#1346)

I assume this is a typo?

@adamalton can you also look into this logger running during testing in CI?

https://github.com/web3-storage/web3.storage/runs/6553500624?check_suite_focus=true

* fix: db migrations versioning (#1375)

* chore(main): release api 6.1.0 (#1356)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* chore(main): release cron 1.1.0 (#1357)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* docs: note db schema required in api readme

...and tweak website README to note that you are using a mock API

* chore: add test for CORS OPTIONS handler (#1331)

* fix: show custom storage quota to user (#1338)

* chore: fix tags in api user info (#1379)

* chore(main): release api 6.1.1 (#1382)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* chore(main): release website 2.5.0 (#1348)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* chore: change the storage cron job config to run in prod, and every 6 hours (#1371)

* chore: trigger crons workflow on conf change (#1394)

* feat: Adding HasDeleteRestriction user_tag (#1390)

* Adding the type and failing HTTP DELETE operations if this tag is set.
* See nftstorage/admin.storage#66

* fix: clone env so new each request (#1396)

* chore(main): release api 6.2.0 (#1397)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* feat: DB schema and API for user_tag_proposal. (#1006)

* Users create records in this table and admins manage it.

Co-authored-by: trigramdev9 <jsdevel@trigram.co>

* feat: Adding admin ability to search by github_id (#1403)

* See nftstorage/admin.storage#68

* chore: optimise getUserByStorage query (#1405)

* fix: optimise getUserByStorage query to avoid timeouts (#1412)

* chore: update SQL migration to drop the old `users_by_storage_used` function before replacing (#1414)

* fix: update incorrect dag sizes job (#1059) (#1196)

* fix: 404 API http reference links (#1358)

Fixes: #1359

* fix(http docs): incorrect endpoint in description (#1429)

* fix(http docs): incorrect endpoint in description

* test: trigger a rebuild via CI

Co-authored-by: orun <orun@agencyundone.com>

* chore: add Wrangler worker env for Josh (#1351)

* chore: Add Wrangler worker env for Josh.

* chore: Add GATEWAY_URL to josh wrangler config.

* chore: Add default `GATEWAY_URL` var to wrangler template.

* chore(main): release website 2.5.1 (#1417)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix: docs toc highlight on click (#1392)

* fix: toc highlight on click - docs

* fix: Replace document query with inline conditional classes and tweak scroll magic scene settings

* test: force rebuild

Co-authored-by: svvimming <benmunozj@gmail.com>
Co-authored-by: orun <orun@agencyundone.com>

* fix: refactor accordion content (#1391)

* fix: refactor accordion content

* style: reduce mobile padding below faq accordion

Co-authored-by: orun <orun@agencyundone.com>

* fix: Adding user_tag_proposal schema to reset.sql for local dev (#1445)

* Introduced in #1006

* feat: Split file manager table into uploaded & pinned (#1363)

* feat: Filemanager header file type tabs and split table by type

* fix: Storage manager progress bar double border & height

* feat: File manager search results in title + margins and spacing

* test: Console log all files

* test: Console log files and user storage data

* feat: Mock data for files using pinning service + api request for pinned files

* test: Fetch pinning data

* feat: Add loading state to pins tab table

* feat: Add loading state to pins tab table

* feat: Add uploaded/pinned tab url param

* test: Pinned fetch request

* test: Pinned fetch request

* revert 'test: Pinned fetch request'

* revert 'test: Pinned fetch request'

* test: Pinning fetch request with generated token

* revert 'test: Pinning fetch request with generated token'

* test: Pass generated API token to /pins GET request

* test: console log API

* feat: Disable pins table if no files are present & revert pins status to pinned in request

* chore: fix linting warning

* style: Responsive file manager header layout at small breakpoints

* feat: Account page UI minor tweaks

* feat: reduce font size (#1411)

* feat: Message bar incident/maintenance name (#1335)

* feat: Docs automatically generated files

* refactor: Message banner incident/maintenance message displays name

* chore: Remove testing materials

* chore: Remove testing materials

* Revert "chore: Remove testing materials"

This reverts commit ab5f9e9.

* Revert "refactor: Message banner incident/maintenance message displays name"

This reverts commit 7359dbe.

* Revert "feat: Docs automatically generated files"

This reverts commit 26ad9ca.

* refactor: Message banner incident/maintenance message displays name

* chore: Remove testing materials

* chore(main): release cron 1.1.1 (#1413)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix: use NODE_TLS_REJECT_UNAUTHORIZED=0 env var for storage cron job (#1418)

This allows the direct connection to the Postgres DB to work.

* chore: add package-lock change report to PRs (#1453)

Bot to add comment on PRs with a human readable report of changes to package-lock.json

see: https://github.com/marketplace/actions/npm-lockfile-changes

License: (Apache-2.0 AND MIT)
Signed-off-by: Oli Evans <oli@tableflip.io>

* docs: clarify payload size limit for /car endpoint (#1457)

* chore: get back integrity and resolved keys for deps (#1456)

* chore: only check package-lock diff on PRs (#1460)

only check package-lock diff on PRs

see: #1453
which produces nice package-lock reports for our PRs but errors when run from not-a-pr.

License: (Apache-2.0 AND MIT)
Signed-off-by: Oli Evans <oli@tableflip.io>

* fix: Adjust progress bar styles for pinned files minimum cases

Co-authored-by: Joanna Ong <ongjfu@gmail.com>
Co-authored-by: Adam Alton <adamalton@gmail.com>
Co-authored-by: Paolo Chillari <flea89@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
Co-authored-by: e-schneid <99349687+e-schneid@users.noreply.github.com>
Co-authored-by: Gary Homewood <gary@potatolondon.com>
Co-authored-by: Paolo <paolo@potatolondon.com>
Co-authored-by: Oli Evans <oli@tableflip.io>
Co-authored-by: francois-potato <103063611+francois-potato@users.noreply.github.com>
Co-authored-by: Vasco Santos <santos.vasco10@gmail.com>
Co-authored-by: Josh Jarvis <josh.jarvis@potatolondon.com>
Co-authored-by: Joe Spencer <js.developer.undefined@gmail.com>
Co-authored-by: Joe Spencer <jsdevel@trigram.co>
Co-authored-by: Jorropo <jorropo.pgm@gmail.com>
Co-authored-by: Yusef Napora <yusef@napora.org>
Co-authored-by: orun <orun@agencyundone.com>
Co-authored-by: Hugo Dias <hugomrdias@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enforcing Web3 user storage capacity limits - first iteration
5 participants