Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initialization phase to v2 file reader (and re-enable pushdown filtering) #2860

Open
Tracked by #2856
westonpace opened this issue Sep 11, 2024 · 0 comments
Open
Tracked by #2856

Comments

@westonpace
Copy link
Contributor

Pushdown filtering was prototyped but never fully implemented. One of the challenges is that pushdown filtering requires an "initialization" phase which could be combined as part of opening the file or separate. The reasoning for making it separate is that we might not know, at file open, which fields we need to access and use in filters.

We also need to introduce some kind of metadata cache that schedulers / decoders have access to. This would be populated on initialization (or first read) and could be reused for future reads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant