Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine Cargo workspaces #23

Merged
merged 6 commits into from
Apr 23, 2021
Merged

Conversation

andygrove
Copy link
Member

  • Ballista is now part of the default workspace
  • Updated Ballista version from 0.4.2-SNAPSHOT to 0.5.0-SNAPSHOT since the next release is significant as first from the new repo
  • I did not change the DataFusion version and we will need to do that at some point (4.1.0 or 5.0.0 next?)

Closes #17

@andygrove
Copy link
Member Author

@alamb @jorgecarleitao Are you ok with this approach of having a single Cargo workspace for all crates? The dependency is still that Ballista depends on DataFusion.

@Dandandan This also removes snmalloc as a default feature so that we don't require CI and all DataFusion devs to compile with snmalloc. My thinking is that it makes sense to build Ballista executors with snmalloc when creating the Docker images. Let me know what you think.

@codecov-commenter
Copy link

codecov-commenter commented Apr 22, 2021

Codecov Report

Merging #23 (f95bd80) into master (c365a4f) will increase coverage by 5.42%.
The diff coverage is 82.20%.

❗ Current head f95bd80 differs from pull request most recent head f961577. Consider uploading reports for the commit f961577 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master      #23      +/-   ##
==========================================
+ Coverage   70.41%   75.83%   +5.42%     
==========================================
  Files         123      135      +12     
  Lines       21261    23152    +1891     
==========================================
+ Hits        14970    17557    +2587     
+ Misses       6291     5595     -696     
Impacted Files Coverage Δ
ballista/rust/client/src/context.rs 0.00% <0.00%> (ø)
ballista/rust/core/src/datasource.rs 0.00% <ø> (ø)
ballista/rust/core/src/error.rs 0.00% <ø> (ø)
...sta/rust/core/src/serde/logical_plan/from_proto.rs 40.56% <ø> (+40.56%) ⬆️
...llista/rust/core/src/serde/scheduler/from_proto.rs 0.00% <ø> (ø)
ballista/rust/core/src/serde/scheduler/mod.rs 14.28% <ø> (+14.28%) ⬆️
ballista/rust/core/src/serde/scheduler/to_proto.rs 0.00% <ø> (ø)
datafusion/src/optimizer/projection_push_down.rs 98.66% <ø> (ø)
datafusion/src/physical_plan/hash_utils.rs 100.00% <ø> (+2.89%) ⬆️
datafusion/src/optimizer/constant_folding.rs 91.95% <33.33%> (-0.36%) ⬇️
... and 70 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c365a4f...f961577. Read the comment docs.

@andygrove andygrove mentioned this pull request Apr 22, 2021
@andygrove andygrove added ballista datafusion Changes in the datafusion crate labels Apr 22, 2021
@andygrove andygrove self-assigned this Apr 22, 2021
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, I think this approach of a single workspace is fine, as I expect most time people who are working on only datafusion can run cargo test -p datafusion and avoid the build of ballista if they so desire.

Copy link
Member

@jorgecarleitao jorgecarleitao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I understood that a cargo workspace is just a way to have a common dependency resolution tree (a common Cargo.lock for all projects), see here, and some other smaller things.

I have not yet fully understood the consequences of this observation to conclude whether a workspace for all projects is useful for us or not; just a neutral information / observation. ^_^

@andygrove andygrove merged commit 74cdf6f into apache:master Apr 23, 2021
@andygrove andygrove deleted the combine-workspaces branch April 23, 2021 12:37
@houqp houqp added the development-process Related to development process of DataFusion label Jul 29, 2021
andygrove added a commit that referenced this pull request Jan 12, 2023
* Initial commit

* initial commit

* failing test

* table scan projection

* closer

* test passes, with some hacks

* use DataFrame (#2)

* update README

* update dependency

* code cleanup (#3)

* Add support for Filter operator and BinaryOp expressions (#4)

* GitHub action (#5)

* Split code into producer and consumer modules (#6)

* Support more functions and scalar types (#7)

* Use substrait 0.1 and datafusion 8.0 (#8)

* use substrait 0.1

* use datafusion 8.0

* update datafusion to 10.0 and substrait to 0.2 (#11)

* Add basic join support (#12)

* Added fetch support (#23)

Added fetch to consumer

Added limit to producer

Added unit tests for limit

Added roundtrip_fill_none() for testing when None input can be converted to 0

Update src/consumer.rs

Co-authored-by: Andy Grove <andygrove73@gmail.com>

Co-authored-by: Andy Grove <andygrove73@gmail.com>

* Upgrade to DataFusion 13.0.0 (#25)

* Add sort consumer and producer (#24)

Add consumer

Add producer and test

Modified error string

* Add serializer/deserializer (#26)

* Add plan and function extension support (#27)

* Add plan and function extension support

* Removed unwraps

* Implement GROUP BY (#28)

* Add consumer, producer and tests for aggregate relation

Change function extension registration from absolute to relative anchor
(reference)

Remove operator to/from reference

* Fixed function registration bug

* Add test

* Addressed PR comments

* Changed field reference from mask to direct reference (#29)

* Changed field reference from masked reference to direct reference

* Handle unsupported case (struct with child)

* Handle SubqueryAlias (#30)

Fixed aggregate function register bug

* Add support for SELECT DISTINCT (#31)

Add test case

* Implement BETWEEN (#32)

* Add case (#33)

* Implement CASE WHEN

* Add more case to test

* Addressed comments

* feat: support explicit catalog/schema names in ReadRel (#34)

* feat: support explicit catalog/schema names in ReadRel

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: use re-exported expr crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move files to subfolder

* RAT

* remove rust.yaml

* revert .gitignore changes

* tomlfmt

* tomlfmt

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Daniël Heres <danielheres@gmail.com>
Co-authored-by: JanKaul <jankaul@mailbox.org>
Co-authored-by: nseekhao <37189615+nseekhao@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
alamb pushed a commit that referenced this pull request Jul 23, 2024
* Configurable date field extraction style for unparsing (#21)

* Add support for IntervalStyle::MySQL (#18)

* Support alternate format for Int64 unparsing (SIGNED for MySQL) (#22)

* Alternate format support for Timestamp casting (DATETIME for MySQL) (#23)

* Improve

* Fix clippy and docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate development-process Related to development process of DataFusion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Ballista to default cargo workspace
5 participants