Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collection and index updates and fixes #362

Merged
merged 9 commits into from
May 27, 2020
Merged

Collection and index updates and fixes #362

merged 9 commits into from
May 27, 2020

Conversation

sanderpick
Copy link
Member

@sanderpick sanderpick commented May 27, 2020 •
edited
Loading

Closes #291
Closes #359
Closes #360
Closes #361

DB

  • Adds UpdateCollection and associated API+client methods
  • Adds DeleteCollection and associated API+client methods
  • Adds GetCollectionIndexes and associated API+client methods
  • Adds dropIndex to Collection method
  • Adds index validation:
    • Index path must hit a valid type in the schema
    • Supported types: string, number, integer, boolean
    • For unique indexes, ensure collection does not contain multiple instances with the same value at path
  • Better schema validation errors

Some things to consider:

  • Updating a collection schema could cause old data to be unreadable if the particular API doesn't have access to the raw JSON results. This can happen when I field is renamed.
  • Along those lines, there's still work to do here on migrations. A migration config could go along with a collection update request, which would handle field re-mapping. Realm could be some inspiration here: Schema migrations #363
  • Back-filling / re-indexing is still not a thing: Backfill index on creation #286

sanderpick added 2 commits May 26, 2020 19:48
@sanderpick
db: cleanup indexing, add update collection method
21050c9 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick
db: adds delete collection, more index cleanup
499d507 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick sanderpick self-assigned this May 27, 2020
@sanderpick
db: add update collection api method and more tests
5794941 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick sanderpick force-pushed the sander/update-collections branch from 6e093f4 to 5794941 Compare May 27, 2020 18:12
sanderpick added 2 commits May 27, 2020 11:48
@sanderpick
db: add get indexes method to api
106ac88 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick
db: ensure unique indexes can be unique
38ae431 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick sanderpick marked this pull request as ready for review May 27, 2020 19:38
sanderpick added 2 commits May 27, 2020 13:11
@sanderpick
db: use token opt in collection methods
1cff531 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick
db: use two locks
8281496 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
sanderpick
sanderpick commented May 27, 2020
View reviewed changes
db/collection.go Show resolved Hide resolved
db/collection.go Show resolved Hide resolved
db/collection.go Show resolved Hide resolved
db/db.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
@sanderpick sanderpick requested review from jsign and carsonfarmer May 27, 2020 21:00
sanderpick
sanderpick commented May 27, 2020
View reviewed changes
db/db.go Show resolved Hide resolved
@sanderpick
db: check index path on hydration
014b336 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
carsonfarmer
carsonfarmer approved these changes May 27, 2020
View reviewed changes
Copy link
Member

@carsonfarmer carsonfarmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I'm pleased to see how much of the old indexing functionality was still useful! Overall a much cleaner usage, and exposing these APIs via the clients will also help quite a bit with access from clients etc. No nits from here, very nice to see the docs updates 👍 .

api/client/client.go Show resolved Hide resolved
api/client/client_test.go Show resolved Hide resolved
db/collection_test.go Show resolved Hide resolved
db/collection_test.go Show resolved Hide resolved
db/collection_test.go Show resolved Hide resolved
db/collection_test.go Show resolved Hide resolved
db/db.go Show resolved Hide resolved
db/db.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
db/index.go Show resolved Hide resolved
jsign
jsign approved these changes May 27, 2020
View reviewed changes
Copy link
Contributor

@jsign jsign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!, left some nits/comments.

api/client/client_test.go Show resolved Hide resolved
api/service.go Outdated Show resolved Hide resolved
db/collection.go Outdated Show resolved Hide resolved
db/db.go Show resolved Hide resolved
db/db.go Outdated Show resolved Hide resolved
@sanderpick sanderpick mentioned this pull request May 27, 2020
Open
@sanderpick
review: address feedback
641cc5c 
Signed-off-by: Sander Pick <sanderpick@gmail.com>
@sanderpick
Copy link
Member Author

Thanks for the reviews guys 🍺

@sanderpick sanderpick merged commit 907f0f0 into master May 27, 2020
@sanderpick sanderpick deleted the sander/update-collections branch May 27, 2020 23:15
dgtony pushed a commit to anyproto/go-threads that referenced this pull request Aug 24, 2020
@sanderpick @dgtony
Collection and index updates and fixes (textileio#362)
a556125 
* db: cleanup indexing, add update collection method

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: adds delete collection, more index cleanup

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: add update collection api method and more tests

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: add get indexes method to api

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: ensure unique indexes can be unique

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: use token opt in collection methods

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: use two locks

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* db: check index path on hydration

Signed-off-by: Sander Pick <sanderpick@gmail.com>

* review: address feedback

Signed-off-by: Sander Pick <sanderpick@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsign jsign jsign approved these changes

@carsonfarmer carsonfarmer carsonfarmer approved these changes

Assignees

@sanderpick sanderpick

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@sanderpick @carsonfarmer @jsign