Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement storage limits and status information for users and the system #45

Merged
merged 1 commit into from
Nov 16, 2022
Merged

Implement storage limits and status information for users and the system #45

merged 1 commit into from
Nov 16, 2022

Conversation

tahpot
Copy link
Member

@tahpot tahpot commented Nov 16, 2022

Closes #41

@tahpot tahpot self-assigned this Nov 16, 2022
@tahpot tahpot merged commit b7d4b16 into feature/37-implement-vip-3 Nov 16, 2022
@tahpot tahpot deleted the feature/41-storage-limits branch November 16, 2022 10:32
tahpot added a commit that referenced this pull request Nov 24, 2022
@tahpot
Feature/37 implement vip 3 (#44)
6ff6d4a 
* Add scaffolding for new did storage support. Refactor routes.

* Initial progress on create and get endpoints

* Updates working. Get all versions working.

* Deletion now working. All tests passing.

* Support field validation for create, including proof verification

* Tests destroy database before starting. Fix create and update tests to work with new verification.

* Fix missing versionId field on index

* Upgrade to latest ethers version

* Update error messages. Wrap versions response in a versions parameter.

* Fix handling of all versions being wrapped in a versions parameter.

* Ensure signature verification for update, delete

* Implement storage limits and status information for users and the system (#45)

* Remove redundant hash key

Co-authored-by: Chris <chris@verida.io>
tahpot added a commit that referenced this pull request Jan 14, 2023
@tahpot
Release/2.0 acacia (#33)
0e50140 
* Fix/13 improve auth security (#14)

* Support running a single test or multiple. Add JWT dependency.
* Implement JWT authorization. Use a new consent message for signature verification.
* Update permission tests to support JWT auth.
* Implement JWT tests
* Add README instructions for enabling JWT auth in CouchDb
* Fix incorrect merge issues.
* Add support for generating and verifying refresh token
* More refresh token unit tests. Add access token support. Db refactor.
* Fix permission unit tests with new refactor.
* Update server endpoints to use new auth. Create server integration tests for auth.
* Hash database name server side before creating
* Verify context name when verifying refresh token. Don't generate access token if refresh token is invalid.
* Confirm expected user is admin of database before deleting it.
* Ensure database names begin with "v"
* Return an access token and host when requesting a refresh token.
* Support deleteDatabase endpoint
* Expand unit tests
* Split controller into auth and user.
* Fix broken test
* Fix broken delete database calls
* Remove JWT test no longer required
* Support invalidating device ID
* Invalidating devices now working correctly with tests
* Implement garbage collection
* Fix building DSN for public credentials
* Add CouchDB configuration note around basic auth
* Add more docs
* Fix edge case issues identified during testing
* Fix yarn dependencies
* Fix example .env files and remove duplicate.
* Describe how authentication works
* Rename to authorization
* Add missing account-node dev dependency
* Fix minor issues with tests. Better config docs in README.
* Resolve feedback for further review.
* Fix missed merge issue
* Support running single test
* Add support for saving user databases and getting the info as an authorized user. More HTTP status improvements and better checking for valid request params.
* Add missing require for PouchDB
* Cleanup didsToUsernames
* Destructure some vars
* Add cors requirement to README
* Add isTokenValid endpoint to verify a refresh token is still valid and obtain the expiry.

* Support the new decentralized did-client implementation (#32)

* adding acacia testnet

* Update sample.env

* Fix server tests. Add support for deleting all databases for a user. (#35)

* Update dependencies to acacia rc2

* Update account node dependency for tests

* Fix issue with documenting how to configure HMAC key in couchdb

* Add better comments

* Update config to use testnet storage node

* Add docker and docker-compose (#27)

* Remove body-parser as deprecated

* Add Dockerfile

* Replace require with import for the js work without babel

* Add docker-compose

* Update node to latest 14

* Add gh workflow to build and publish docker image

* Rearrange dockerfile to improve image size

* Uncomment dockerhub

* Add docker info and single node config to readme

* Update to latest verida dependencies. Remove module so tests run.

* Remove lambda support

* Details on how to do the Docker build and push it.

* Update REARME.md: enable_cors = true

* update Docker build command

* Changes to make deployment work. Yarn build must happen in Docker container

* Support signing all responses using the storage node VDA private key. (#42)

* Add support for /user/usage details (#43)

* add logging to make debugging Docker image somewhat possible

* fix logging message

* update README to correct authentication info. Add logging to dbInit exception handler

* logging for failure of /auth/public; clean up pakage.json

* update docs on HASH_KEY not being needed

* Feature/37 implement vip 3 (#44)

* Add scaffolding for new did storage support. Refactor routes.

* Initial progress on create and get endpoints

* Updates working. Get all versions working.

* Deletion now working. All tests passing.

* Support field validation for create, including proof verification

* Tests destroy database before starting. Fix create and update tests to work with new verification.

* Fix missing versionId field on index

* Upgrade to latest ethers version

* Update error messages. Wrap versions response in a versions parameter.

* Fix handling of all versions being wrapped in a versions parameter.

* Ensure signature verification for update, delete

* Implement storage limits and status information for users and the system (#45)

* Remove redundant hash key

Co-authored-by: Chris <chris@verida.io>

* Update README.md 

Remove Docker compose (moved to infrastructure)
Remove redundant value in windows config.

* Force lowercase username when generating usernames

* Add comments about running tests

* Include permissions in database info

* Update dependencies to next release candidate

* delay start of docker image until $DB_HOST reolves

* Add support for a NODE_URI

* typo

* remove checking for network now SN supports different endpoint

* Move signature tests to the end to avoid issues with testing servers with invalid signatures.

* Support internal and external DB hostnames

* Fix hostname generation.

* Fix build external host

* Add console errors and warnings to help with debugging

* Support looking up a DID document and caching it

* Include public key in system status

* Feature/49 server replication (#50)

* Rename applicationName to contextName. Add replicator permissions.

* Add generateHash utility method

* Refactor so context databases are in their own database that will be replicated, instead of all stored in a single database. Other minor fixes and test improvements.

* Implement untested auth/replicationCreds

* Implementation complete, untested.

* Add comments for next steps

* First pass at replication test

* Fix missing endpoint variable

* Bug fixes and enhanced logging

* Bug fixes from testing

* Bug fixes

* Support generating correct CouchURI for server

* Bug fixes

* Fix syntax error

* First tests passing

* Fix clearing databases. Add warnings.

* Ensure replicator user has did context role

* Bug fixes

* Try basic auth header

* Update replication document creation

* Create new replicater user for replication instead of using admin

* Bug fix replication

* Update working tests

* All tests written so far, pass!

* Remove commented test code

* Code cleanup

* Code cleanup. Replication test cleanup. Verify timestamp when fetching replication creds.

* Add comments in .env file

Co-authored-by: Chris <chris@verida.io>

* Feature/49 server replication (#51)

* Rename applicationName to contextName. Add replicator permissions.

* Add generateHash utility method

* Refactor so context databases are in their own database that will be replicated, instead of all stored in a single database. Other minor fixes and test improvements.

* Implement untested auth/replicationCreds

* Implementation complete, untested.

* Add comments for next steps

* First pass at replication test

* Fix missing endpoint variable

* Bug fixes and enhanced logging

* Bug fixes from testing

* Bug fixes

* Support generating correct CouchURI for server

* Bug fixes

* Fix syntax error

* First tests passing

* Fix clearing databases. Add warnings.

* Ensure replicator user has did context role

* Bug fixes

* Try basic auth header

* Update replication document creation

* Create new replicater user for replication instead of using admin

* Bug fix replication

* Update working tests

* All tests written so far, pass!

* Remove commented test code

* Code cleanup

* Code cleanup. Replication test cleanup. Verify timestamp when fetching replication creds.

* Replication tests all passing with remote 3x servers

* Remove console output

Co-authored-by: Chris <chris@verida.io>

* Feature/52 user db list replicated (#54)

* Ensure user databases are replicated

* Add checkReplication tests confirming databases are re-created

* Fix user database issues. Add debug logging.

* Bug fix database creation via checkReplication()

* Basic replication unit tests passing

* Remove excess debug output

* Fix trailing slash issue

* Fix trailing slash issues

* Adding cache checks for DID documents

* Fix database list database incorrect config

* Handle missing DID document

* Throw error if unknown check databse error

* Improve service endpoint checks on checkReplication()

* Ensure user database list database has replicator local role

* Fix issue with user database list database not being recreated. Minor refactor.

* Handle garbage collection where token already deleted

* Handle user database list database document conflict

* Fix caching of couch instance not respecting external v internal

* Add debug output to nodes about cache status

* Add error message when unable to resolve DID

* Add debug logging

* Unify storage node checks and hashes to use the hostname only

* Make endpointUri more resiliant

* Upgrade encryption utils to latest

* Support recoverying from replication credential errors

* Add debug logging

* Add storage node HTTP timeouts. Add more debugging.

* Fix replicater creds to have replicaterUsername has id

* Fix replicater username to only use hostname

* Fix incorrect username being loaded

* Fix replication to start working upon creation of new databases, only update password if it changes, re-use existing password where possible.

* Check replication credentials after all databases have been checked.

* Refactor how endpoint security checks occur. Cleanup console output.

* Improve debug output

* More debug output for testing

* Remove redundant record variable

* Add delete failed replication meethod

* Fix incorrect delete replication code

* More debugging

* More debug output

* Update order of code

* Change order so that failed replication entries are removed first, so they are auto created later.

* Add more debug output for testing=

* Force update of remote credentials if they are definitely invalid

* Add debug output

* Add more debug output

* Fix issue with fetchReplicaterCredential setting the wrong key for the credential cache

* Fix incorrect username

* Restructure how replication fails are fixed

* Add more debug output, fix replication status id parameter

* More debug output

* More debugging

* Fix sample.env comment

* Reduce debug output

* d

* Refactor replication and improve debug output

* Handle unable to fetch DID error

* Add checkreplication debug header

* Lots of bug fixes and refactor

* Bug fix incorrect instance reference

* Force cache refresh if service endpoint not found

* Handle missing did document

* set winning_revs_only: true

* Remove winning_revs_only

* Remove permissions check

* Add debug logging

* More debug output

* Debug to find error

* Support ignore cache option when fetching DID document

* Correctly handle empty DIDs

* Update CHANGELOG in preparation for v2.0.0 release
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tahpot tahpot

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@tahpot