Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill index on creation #286

Open
asutula opened this issue Mar 26, 2020 · 4 comments
Open

Backfill index on creation #286

asutula opened this issue Mar 26, 2020 · 4 comments

Comments

@asutula
Copy link
Member

asutula commented Mar 26, 2020

Currently we start building the index from the time it was created, but we should go though all existing data and index it as well.

@asutula
Copy link
Member Author

asutula commented Mar 27, 2020

Feeling like this might be quite complex. Indexing can take an unbounded amount of time, so that has me thinking it will have to be backed by a durable, persistent, job queue kinda setup.

@asutula
Copy link
Member Author

asutula commented Mar 27, 2020

You'd also want it to be non-blocking

@sanderpick
Copy link
Member

I think most DBs block while doing an initial index (or back filling), e.g., (https://docs.mongodb.com/manual/core/index-creation/), but some of the more modern one (Rethink?) do not. The simpler solution might be a good first step.

@sanderpick
Copy link
Member

Our situation might actually be a bit more dangerous though, since we're handing full admin control over to a dev with a client lib. At least they can only mess up their own DB (once ACLs are a thing).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants