Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RemoteDocumentCache APIs for Indexing #5988

Merged
merged 7 commits into from
Mar 18, 2022
Merged

Conversation

schmidt-sebastian
Copy link
Contributor

@schmidt-sebastian schmidt-sebastian commented Feb 9, 2022

This changes the schema of the Remote Document Cache to rewrite the primary keys. The keys are now:

[ [prefix path ], collection group, read time, document ]

This allows for efficient scanning of indexed and non-indexed queries using getAll(). The PR also adds getDocumentsFromCollectionGroup, which means that all functionality required by Indexing is now available.

TODO:

  • Benchmark the migration
  • Re-write multi tab to use the collectionGroup+ReadTime index

@changeset-bot
Copy link

changeset-bot bot commented Feb 9, 2022

🦋 Changeset detected

Latest commit: 8ce57c0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@firebase/firestore Patch
firebase Patch
@firebase/firestore-compat Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@schmidt-sebastian schmidt-sebastian changed the title Mrschmidt/rdc RemoteDocumentCache APIs for Indexing Feb 10, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Feb 22, 2022

Changeset File Check ✅

  • No modified packages are missing from the changeset file.
  • No changeset formatting errors detected.

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Feb 23, 2022

Size Report 1

Affected Products

  • @firebase/app

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser13.9 kB13.8 kB-133 B (-1.0%)
    esm518.8 kB18.0 kB-738 B (-3.9%)
    main19.7 kB18.9 kB-820 B (-4.2%)
    module13.9 kB13.8 kB-133 B (-1.0%)
  • @firebase/app-check

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser25.1 kB25.2 kB+82 B (+0.3%)
    esm529.7 kB29.8 kB+148 B (+0.5%)
    main30.9 kB31.0 kB+148 B (+0.5%)
    module25.1 kB25.2 kB+82 B (+0.3%)
  • @firebase/auth-compat

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser20.1 kB20.1 kB+1 B (+0.0%)
    esm526.9 kB26.9 kB+1 B (+0.0%)
    main29.5 kB29.5 kB+1 B (+0.0%)
    module20.1 kB20.1 kB+1 B (+0.0%)
  • @firebase/auth/cordova

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser180 kB180 kB+497 B (+0.3%)
    module180 kB180 kB+497 B (+0.3%)
  • @firebase/auth/internal

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser163 kB164 kB+445 B (+0.3%)
    esm5212 kB213 kB+497 B (+0.2%)
    main179 kB180 kB+497 B (+0.3%)
    module163 kB164 kB+445 B (+0.3%)
  • @firebase/auth/react-native

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser164 kB165 kB+497 B (+0.3%)
    module164 kB165 kB+497 B (+0.3%)
  • @firebase/firestore

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser248 kB251 kB+2.72 kB (+1.1%)
    esm5309 kB312 kB+2.93 kB (+0.9%)
    main498 kB502 kB+4.72 kB (+0.9%)
    module248 kB251 kB+2.72 kB (+1.1%)
    react-native249 kB251 kB+2.72 kB (+1.1%)
  • @firebase/firestore-lite

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser73.1 kB73.2 kB+59 B (+0.1%)
    esm586.5 kB86.6 kB+63 B (+0.1%)
    main126 kB126 kB+75 B (+0.1%)
    module73.1 kB73.2 kB+59 B (+0.1%)
    react-native73.3 kB73.4 kB+59 B (+0.1%)
  • @firebase/functions-compat

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser1.67 kB1.68 kB+1 B (+0.1%)
    esm51.83 kB1.83 kB+1 B (+0.1%)
    main2.20 kB2.20 kB+1 B (+0.0%)
    module1.67 kB1.68 kB+1 B (+0.1%)
  • @firebase/installations

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser17.3 kB17.8 kB+484 B (+2.8%)
    esm522.3 kB22.9 kB+628 B (+2.8%)
    main23.1 kB23.7 kB+599 B (+2.6%)
    module17.3 kB17.8 kB+484 B (+2.8%)
  • @firebase/messaging

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser20.8 kB20.8 kB+21 B (+0.1%)
    esm526.1 kB26.2 kB+18 B (+0.1%)
    main26.8 kB26.8 kB-7 B (-0.0%)
    module20.8 kB20.8 kB+21 B (+0.1%)
  • @firebase/messaging-compat

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser2.08 kB2.08 kB+1 B (+0.0%)
    esm52.51 kB2.51 kB+1 B (+0.0%)
    main2.90 kB2.90 kB+1 B (+0.0%)
    module2.08 kB2.08 kB+1 B (+0.0%)
  • @firebase/messaging-sw

    TypeBase (09e6243)Merge (be0bedc)Diff
    main29.6 kB29.6 kB-8 B (-0.0%)
    module22.8 kB22.8 kB+20 B (+0.1%)
  • @firebase/util

    TypeBase (09e6243)Merge (be0bedc)Diff
    browser20.5 kB23.4 kB+2.86 kB (+13.9%)
    esm521.9 kB25.5 kB+3.65 kB (+16.7%)
    main26.7 kB30.5 kB+3.82 kB (+14.3%)
    module20.5 kB23.4 kB+2.86 kB (+13.9%)
  • bundle

    43 size changes

    TypeBase (09e6243)Merge (be0bedc)Diff
    analytics (logEvent)41.8 kB40.0 kB-1.78 kB (-4.3%)
    app-check (CustomProvider)35.7 kB33.7 kB-1.99 kB (-5.6%)
    app-check (ReCaptchaEnterpriseProvider)37.9 kB35.9 kB-1.99 kB (-5.3%)
    app-check (ReCaptchaV3Provider)37.8 kB35.8 kB-1.99 kB (-5.3%)
    auth (Anonymous)65.3 kB63.5 kB-1.77 kB (-2.7%)
    auth (EmailAndPassword)69.3 kB67.6 kB-1.77 kB (-2.6%)
    auth (GoogleFBTwitterGitHubPopup)89.1 kB87.3 kB-1.77 kB (-2.0%)
    auth (GooglePopup)88.9 kB87.1 kB-1.77 kB (-2.0%)
    auth (GoogleRedirect)89.1 kB87.3 kB-1.77 kB (-2.0%)
    auth (Phone)75.3 kB73.6 kB-1.77 kB (-2.3%)
    database (Append to a list of data)146 kB143 kB-2.05 kB (-1.4%)
    database (Filtering data)144 kB142 kB-2.05 kB (-1.4%)
    database (Listen for child events)160 kB158 kB-2.05 kB (-1.3%)
    database (Listen for value events + Detach listeners)160 kB158 kB-2.05 kB (-1.3%)
    database (Listen for value events)160 kB158 kB-2.05 kB (-1.3%)
    database (Read data once)152 kB150 kB-2.05 kB (-1.3%)
    database (Save data as transactions)162 kB160 kB-2.05 kB (-1.3%)
    database (Sort data)146 kB144 kB-2.05 kB (-1.4%)
    database (Write data)145 kB143 kB-2.05 kB (-1.4%)
    firestore (Persistence)261 kB262 kB+706 B (+0.3%)
    firestore (Query Cursors)203 kB202 kB-1.33 kB (-0.7%)
    firestore (Query)204 kB203 kB-1.33 kB (-0.6%)
    firestore (Read data once)193 kB192 kB-1.33 kB (-0.7%)
    firestore (Realtime updates)195 kB194 kB-1.33 kB (-0.7%)
    firestore (Transaction)178 kB176 kB-1.33 kB (-0.7%)
    firestore (Write data)177 kB176 kB-1.33 kB (-0.7%)
    firestore-lite (Query Cursors)68.3 kB66.3 kB-1.95 kB (-2.9%)
    firestore-lite (Query)71.3 kB69.4 kB-1.95 kB (-2.7%)
    firestore-lite (Read data once)55.8 kB53.8 kB-1.95 kB (-3.5%)
    firestore-lite (Transaction)73.1 kB71.2 kB-1.95 kB (-2.7%)
    firestore-lite (Write data)58.6 kB56.6 kB-2.01 kB (-3.4%)
    functions (call)29.6 kB27.6 kB-2.01 kB (-6.8%)
    messaging (send + receive)44.9 kB43.3 kB-1.58 kB (-3.5%)
    performance (trace)49.4 kB47.7 kB-1.78 kB (-3.6%)
    remote-config (getAndFetch)44.1 kB42.3 kB-1.78 kB (-4.0%)
    storage (getBytes)37.9 kB35.9 kB-2.01 kB (-5.3%)
    storage (getDownloadURL)40.0 kB38.0 kB-2.01 kB (-5.0%)
    storage (getMetadata)39.4 kB37.4 kB-2.01 kB (-5.1%)
    storage (list + listAll)38.9 kB36.9 kB-2.01 kB (-5.2%)
    storage (updateMetadata)39.7 kB37.7 kB-2.01 kB (-5.1%)
    storage (uploadBytes)44.3 kB42.2 kB-2.01 kB (-4.5%)
    storage (uploadBytesResumable)53.7 kB51.7 kB-2.01 kB (-3.7%)
    storage (uploadString)44.5 kB42.5 kB-2.01 kB (-4.5%)

  • firebase

    24 size changes

    TypeBase (09e6243)Merge (be0bedc)Diff
    firebase-analytics-compat.js26.0 kB24.3 kB-1.70 kB (-6.5%)
    firebase-analytics.js107 kB105 kB-2.23 kB (-2.1%)
    firebase-app-check-compat.js22.7 kB22.7 kB+6 B (+0.0%)
    firebase-app-check.js89.9 kB90.0 kB+125 B (+0.1%)
    firebase-app-compat.js28.2 kB26.4 kB-1.85 kB (-6.5%)
    firebase-app.js84.2 kB81.4 kB-2.77 kB (-3.3%)
    firebase-auth-compat.js123 kB123 kB+228 B (+0.2%)
    firebase-auth-cordova.js462 kB463 kB+931 B (+0.2%)
    firebase-auth-react-native.js486 kB491 kB+5.73 kB (+1.2%)
    firebase-auth.js411 kB412 kB+667 B (+0.2%)
    firebase-compat.js777 kB778 kB+1.34 kB (+0.2%)
    firebase-firestore-compat.js300 kB303 kB+2.66 kB (+0.9%)
    firebase-firestore-lite.js250 kB250 kB+81 B (+0.0%)
    firebase-firestore.js818 kB822 kB+4.87 kB (+0.6%)
    firebase-functions-compat.js7.95 kB7.95 kB+1 B (+0.0%)
    firebase-messaging-compat.js38.0 kB36.4 kB-1.56 kB (-4.1%)
    firebase-messaging-sw.js102 kB101 kB-1.67 kB (-1.6%)
    firebase-messaging.js101 kB99.2 kB-1.67 kB (-1.7%)
    firebase-performance-compat.js30.8 kB29.1 kB-1.69 kB (-5.5%)
    firebase-performance-standalone-compat.es2017.js87.5 kB86.5 kB-993 B (-1.1%)
    firebase-performance-standalone-compat.js65.7 kB64.0 kB-1.68 kB (-2.6%)
    firebase-performance.js119 kB117 kB-2.23 kB (-1.9%)
    firebase-remote-config-compat.js27.5 kB25.8 kB-1.70 kB (-6.2%)
    firebase-remote-config.js109 kB106 kB-2.23 kB (-2.1%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/XdxN9u04tq.html

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Feb 23, 2022

Size Analysis Report 1

This report is too large (782,147 characters) to be displayed here in a GitHub comment. Please use the below link to see the full report on Google Cloud Storage.

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/RCAhtRIu7I.html

@schmidt-sebastian schmidt-sebastian force-pushed the mrschmidt/rdc branch 2 times, most recently from 80573ca to 5e1a25b Compare February 28, 2022 21:46
Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not reviewed tests yet, but want to provide some feedback first.

Main concern is some places are not very readable.

Also, it looks like we might need to do this for LevelDb as well?

return store.delete(key);
const path = documentKey.path.toArray();
return store.delete([
path.slice(0, path.length - 2),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are quite some places using length - n to get what we want, maybe we should put all these in DocumentKey instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if it will fit well in DocumentKey, as we don't really have good terminology for "path that leads up to the collection". I will figure out how to make this cleaner in place.

"@firebase/firestore": patch
---

The format of some of the IndexedDB data changed. This increases the performance of document lookups after an initial migration. If you do not want to migrate data, you can call `clearIndexedDbPersistence()` before invoking `enablIndexedDbPersistence()`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in line 5, needs another 'e', should be enableIndexedDbPersistence().

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@egilmorez ... Talked to sebastian, and these changes are internal-only, no user-breaking effects. Thanks for adding me to the review.

@schmidt-sebastian ... Please feel free to include me in Firestore PRs, all platforms, especially the PRs that touch public-facing reference docs. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@wu-hui wu-hui assigned schmidt-sebastian and unassigned wu-hui Mar 18, 2022
@schmidt-sebastian schmidt-sebastian merged commit 69aa7b0 into master Mar 18, 2022
@schmidt-sebastian schmidt-sebastian deleted the mrschmidt/rdc branch March 18, 2022 15:49
@google-oss-bot google-oss-bot mentioned this pull request Mar 22, 2022
@firebase firebase locked and limited conversation to collaborators Apr 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants