Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: move parquet related functionality to datafusion-parquet crate #11188

Closed
wants to merge 2 commits into from

Conversation

devinjdangelo
Copy link
Contributor

Which issue does this PR close?

related to #11182

Rationale for this change

Splitting FileFormat implementations out of datafusion-core will help ensure datafusion remains modular and extensible, as well as improve maintainability. ``

What changes are included in this PR?

Early work on moving parquet related code to a datafusion-parquet crate.

Are these changes tested?

No (not complete yet)

Are there any user-facing changes?

Yes

@github-actions github-actions bot added the core Core DataFusion crate label Jul 1, 2024
@alamb
Copy link
Contributor

alamb commented Jul 1, 2024

Nice -- thanks @devinjdangelo

I was thinking about potential ways to organize the crates. One potential in my mind was

datafusion-catalog (has TableProvider, CatalogProvider, etc, maybe Mem*Provider)
datafusion-catalog-listing (ListingTable)
datafusion-datasource-parquet
datafusion-datafsource-avro
datafusion-datasource-csv
datafusion-datasource-json
datafusion-datasource-arrow

Though perhaps that might be overkill (especially for formats like datafusion-datasource-csv...)

Maybe it would be better like

datafusion-catalog (has TableProvider, CatalogProvider, etc, maybe Mem*Provider)
datafusion-catalog-listing (ListingTable)
datafusion-datasource (built in formats like avro, csv, json, arrow, parquet)

Copy link

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the Stale PR has not had any activity for some time label Aug 31, 2024
@github-actions github-actions bot closed this Sep 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate Stale PR has not had any activity for some time
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants