Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

datastore-idb .query doesn't work as expected #198

Open
achingbrain opened this issue Jun 2, 2020 · 1 comment
Open

datastore-idb .query doesn't work as expected #198

achingbrain opened this issue Jun 2, 2020 · 1 comment
Labels
kind/bug A bug in existing code (including security flaws)

Comments

@achingbrain
Copy link
Member

achingbrain commented Jun 2, 2020 •
edited
Loading

store.query opens an IndexedDB transaction and uses a cursor to iterate over each entry in the datastore. The transaction closes automatically when the microtask queue is emptied and there's no outstanding work for it to do, even if the cursor is not at the end of the dataset.

Each query result is yielded to the caller which if they do any async work can cause the transaction to close and the next call to cursor.continue() to throw ERROR: DOMException{stack: 'Error: Failed to execute 'continue' on 'IDBCursor': The transaction is not active..

https://www.npmjs.com/package/idb/v/5.0.2#transaction-lifetime
@achingbrain achingbrain added the need/triage Needs initial labeling and prioritization label Jun 2, 2020
@achingbrain
Copy link
Member Author

You could reopen the cursor at the next index if it fails to continue, but there's no guarantee the caller hasn't mutated the datastore since the transaction was closed so the index may not be consistent.

@achingbrain achingbrain mentioned this issue Jun 3, 2020
Merged
@achingbrain achingbrain linked a pull request Jun 9, 2020 that will close this issue
fix: wrap operations in transactions ipfs/js-datastore-idb#7
Closed
achingbrain referenced this issue in ipfs/js-ipfs-repo Jun 15, 2020
@achingbrain
feat: use datastore-level in the browser again
f2e499c 
level-js seems to handle datastore mutations during queries better
which we need for datastore migrations.

Revist once https://github.com/ipfs/js-datastore-idb/issues/6 is
resolved.
This was referenced Jun 15, 2020
Merged
Closed
achingbrain referenced this issue in ipfs/js-ipfs-repo Jun 15, 2020
@achingbrain
feat: use datastore-level in the browser again (#236)
33663b3 
`level-js` seems to handle datastore mutations during queries better
which we need for datastore migrations.

Revisit once https://github.com/ipfs/js-datastore-idb/issues/6 is
resolved.
@autonome autonome added kind/bug A bug in existing code (including security flaws) and removed need/triage Needs initial labeling and prioritization labels Jul 2, 2020
@achingbrain achingbrain mentioned this issue Dec 13, 2020
Closed
64 tasks
achingbrain referenced this issue in ipfs/js-datastore-idb Mar 14, 2023
@achingbrain
feat!: convert to typescript
a2f3e06 
- Converts module to typescript
- Works around #6 by loading the query results in memory before yielding results
- Publishes as ESM-only
- Updates all deps to latest versions

The query performance will be bad, but at least it will be consistent.

Queries are only run by ipfs/helia during garbage collection which you don't normally do in the browser so it may not be as
bad as it seems.

BREAKING CHANGE: this module has been converted to typescript, updated to the latest interface-datastore version and is
ESM-only
@achingbrain achingbrain mentioned this issue Mar 14, 2023
Merged
achingbrain referenced this issue in ipfs/js-datastore-idb Mar 14, 2023
@achingbrain
feat!: convert to typescript (#98)
a34262f 
- Converts module to typescript
- Works around #6 by loading the query results in memory before yielding results
- Publishes as ESM-only
- Updates all deps to latest versions

The query performance will be bad, but at least it will be consistent.

Queries are only run by ipfs/helia during garbage collection which you don't normally do in the browser so it may not be as bad as it seems.

Read/write performance is much better than `datastore-level` in the browser so the tradeoff may be worth it.

```
┌─────────┬────────────────┬────────┬───────┬───────┐
│ (index) │ Implementation │ ops/s  │ ms/op │ runs  │
├─────────┼────────────────┼────────┼───────┼───────┤
│    0    │   'idb put'    │ 103570 │ 0.01  │ 51785 │
│    1    │  'level put'   │ 55456  │ 0.02  │ 27728 │
└─────────┴────────────────┴────────┴───────┴───────┘
```

BREAKING CHANGE: this module has been converted to typescript, updated to the latest interface-datastore version and is ESM-only
github-actions bot referenced this issue in ipfs/js-datastore-idb Mar 14, 2023
@semantic-release-bot
chore(release): 2.0.0 [skip ci]
af8607e 
## [2.0.0](v1.1.0...v2.0.0) (2023-03-14)

### ⚠ BREAKING CHANGES

* this module has been converted to typescript, updated to the latest interface-datastore version and is ESM-only

### Features

* convert to typescript ([#98](#98)) ([a34262f](a34262f)), closes [#6](https://github.com/ipfs/js-datastore-idb/issues/6)

### Documentation

* fix link ([417af5b](417af5b))

### Trivial Changes

* **deps-dev:** bump aegir from 22.1.0 to 24.0.0 ([f71f236](f71f236))
* **deps-dev:** bump aegir from 24.0.0 to 25.0.0 ([2479fe6](2479fe6))
* Update .github/workflows/stale.yml [skip ci] ([c113be2](c113be2))
* update aegir ([5f00f1b](5f00f1b))
@achingbrain achingbrain mentioned this issue Mar 14, 2023
Closed
@achingbrain achingbrain transferred this issue from ipfs/js-datastore-idb Mar 23, 2023
@achingbrain achingbrain changed the title .query doesn't work as expected datastore-idb .query doesn't work as expected Mar 23, 2023
@hugomrdias hugomrdias removed their assignment Mar 23, 2023
@ipfs ipfs deleted a comment from welcome bot Apr 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: wrap operations in transactions ipfs/js-datastore-idb
3 participants
@autonome @hugomrdias @achingbrain