Release DataFusion 44.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/43.0.0 November 8th, 2024 so next major release would be around December 8, 2024

• Voting Thread: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

Prior release tickets:

• 43.0.0 Release DataFusion 43.0.0 #12470

Items to possibly fix before release

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• [Epic] A Collection of Additional UTF8View support tickets #13504

Improved upgrade experience:

• Make migration to Signature::nullary in 44.0.0 easier / less confusing #13763
• Make DocBuilder migration in 44.0.0 easier #13764
• Making the recursive dependency an optional feature #13766
• Deprecate ScalarUDFImpl::return_type #13717

Downstream project testing:

• datafusion python: chore: Upgrade to DataFusion 44 datafusion-python#972
• comet: Test DataFusion 44.0.0 with Comet #13835
• delta.rs: Test DataFusion 44.0.0 with delta.rs #13834
• Test DataFusion 44.0.0 with Sail #13855

[alamb]

I would love to get this one in:

• Perf: Allow User defined functions to potentially reuse their argument arrays (to avoid new allocations) #13516

Some things to highlight (that are already merged)

• Support vectorized append and compare for multi group by #12996 from @Rachelint as well

[alamb]

Also this hopefully:

• Add SessionConfig reference to ScalarFunctionArgs #13519

[alamb]

@Omega359 and @andygrove suggested #13525 (comment) that for this release we

we file tickets as part of every release to test DF with latest versions of common external dependencies (datafusion-python, iceberg, lance, delta, etc) as part of the gating criteria for a release. Perhaps this would be as simple as changing the DF version in those projects and running their test suite. Any blockers that are found in DF have tickets filed and are fixed, other blockers in the respective projects have tickets filed in those projects

Perhaps for 44.0.0 we can try with own subprojects (Ballista, Comet, DF Python, DF Ray)

[alamb]

I started gathering a list of items we think we should fix prior to the release in the description

[alamb]

Does anyone have any opinion about holding the DataFusion 44 release for the next major arrow release?

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

That would fix a bunch of the StringView issues, but would likely delay DataFusion 44 for a few more weeks

[Omega359]

It's soon to be the holiday season so I'm all for cooking the release a bit more.

[alamb]

I would personally love to see DataFusion 44.0.0 be lauded as "super stable" and have few upgrade issues (we would largely achieve this by testing upgrades with other projects prior to release)

The downside is that the changeset in DataFusion 44.0.0 might be larger

[andygrove]

I'd like to see an upgrade guide for the 44 release (and am willing to take the lead on this). I am trying to upgrade Comet now and am running into some issues.

I am +1 for waiting for the next Arrow release.

[andygrove]

I filed #13702

[andygrove]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

[alamb]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

In my opinion, we would ideally do the testing before we make the release candidate (as the overhead of making RCs is non trivial). However, I am not opposed to extending the voting timeline if that would get us more testing time

[andygrove]

Testing before we create the RC makes sense

[andygrove]

I'm still working on updating Comet to work with latest DF from main (upgrading from 43.0.0). This is quite a disruptive upgrade for us so far.

[alamb]

I have been thinking about this upgrade and the next arrow release

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

Specifically I think we need this upgrade in order to resolve some of the issues in 44 related to string view

• [Epic] A Collection of Additional UTF8View support tickets #13504

However, what I would like to propose is:

1. Plan to release DataFusion 44.0.0 without also upgrading arrow-rs

The downside of this plan is

1. it will take longer to get the arrow upgrade (and this the string view fixes) throughout the ecosystem

The upsides are

1. It will reduce the complexity of the DataFusion 44 upgrade
2. it will give us a longer period to test the upgrade integration DataFusion

If someone is critically waiting on the string view fixes, we can also contemplate making another arrow incremental release

[alamb]

I am also going to coordinate with the delta-rs folks to try and test upgrading prior to release:

• chore: upgrade to datafusion 43 delta-io/delta-rs#2886 (comment)

[timsaucer]

Starting an issue and branch in datafusion-python : apache/datafusion-python#972

[rluvaton]

I'll give my 2 cents from my experience regarding how other popular project handle this issue

In Node.js (I'm a core collaborator) before each release we run the tests of the most popular packages with the new node version and making sure we are not breaking them, if we are we let them know. this is really helpful to avoid breaking the entire world

this process is a CITGM, you can watch this talk for more info

[andygrove]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

[mbwhite]

Hi - would like to get #13647 out as soon as is practical - a small fix but does unblock my use of
DataFusion for Substrait.

Thanks!

[alamb]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

• I have filed Release arrow-rs / parquet minor version 53.4.0 (Jan 2025) arrow-rs#6887 to track this potential release

[alamb]

Just for what it's worth, I appreciate smaller releases more often compared to bigger ones less often, even if it means more more times I need to fix downstream stuff due to breaks, presumably it usually also means less big breaks. And also faster time to get my and other's improvements and fixes into test/use, and thus faster finding of unintended regressions when they happen. That said, I appreciate there's a bunch of factors including the work needed etc, just wanted to offer my 2 cents 😄)

I agree -- thank you @Blizzara

In this case I think we are running into bandwidth / maintainer limitations -- I would really like to make sure our releases are less jarring

Does anyone have time to help test with delta.rs?

• Test DataFusion 44.0.0 with delta.rs #13834

[shehabgamin]

Just for what it's worth, I appreciate smaller releases more often compared to bigger ones less often, even if it means more more times I need to fix downstream stuff due to breaks, presumably it usually also means less big breaks. And also faster time to get my and other's improvements and fixes into test/use, and thus faster finding of unintended regressions when they happen. That said, I appreciate there's a bunch of factors including the work needed etc, just wanted to offer my 2 cents 😄)

I agree -- thank you @Blizzara

In this case I think we are running into bandwidth / maintainer limitations -- I would really like to make sure our releases are less jarring

Does anyone have time to help test with delta.rs?

* [Test DataFusion 44.0.0 with delta.rs #13834](https://github.com/apache/datafusion/issues/13834)

https://github.com/lakehq/sail has imported over 3,800 tests from the Spark code base. With each DataFusion upgrade, we have found that any bugs/breaks likely cause one of our tests to fail. If helpful, we're happy to test DataFusion with Sail and include Sail as part of the testing process for each release!

[findepi]

@shehabgamin that would be awesome, thanks!
It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

[shehabgamin]

@findepi Great will do! Also, over time, we're happy to work on porting any tests into DataFusion that would make sense to have in DataFusion.

We'll integrate Sail with the latest commit in the DataFusion main branch and report back our findings. When should Sail complete testing by? We do have bandwidth this week and next week. We’d be happy to prioritize testing this on our side, just lemme know expectations around timing!

[ozankabak]

It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

Agreed, let's do this

[alamb]

We'll integrate Sail with the latest commit in the DataFusion main branch and report back our findings. When should Sail complete testing by? We do have bandwidth this week and next week. We’d be happy to prioritize testing this on our side, just lemme know expectations around timing!

I am personally hoping we are ready to make an RC from main by early next week

It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

Agreed, let's do this

100% agree -- a good place to start might be any tests that you find fail in your project but pass all DataFusion tests

[andygrove]

I am personally hoping we are ready to make an RC from main by early next week

Just fyi, I may not be online much next week, not that I need to be involved in cutting the RC, but will likely have time the following week to help, if needed.

[alamb]

What I plan to do is try and clean up any remaining outstanding PRs and get them merged over the next few days and I'll try and prepare the RC asap

Any help people can give to help figure out how to get the approved PRs merged would be super helpful:

• https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved+-is%3Adraft

[shehabgamin]

What I plan to do is try and clean up any remaining outstanding PRs and get them merged over the next few days and I'll try and prepare the RC asap

Any help people can give to help figure out how to get the approved PRs merged would be super helpful:

* https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved+-is%3Adraft

Given that many changes will be merged in after pre-release testing, I think it makes sense to have people run their tests one more time once there is a release candidate. This should be effortless since further breaking changes should be minimal or non-existent. Otherwise, this defeats the purpose of pre-release testing because we cannot guarantee the stability of the final release without validating the accumulated changes.

I'm getting started now with testing on Sail. Using the current latest commit from main branch:5d563d9. Work is being done here lakehq/sail#335. Once the code is able to build, a test report will be generated automatically on the PR.

EDIT: Adding #13855

[shehabgamin]

Test complete: #13855 (comment)
Will re-test once there is a release candidate.

[alamb]

Status update here is I think we are looking good to make a RC candidate, either later today or tomorrow

Ease upgrades:

Summary: From my perspective the following issues are nice to have, rather than required for the release

• Make migration to Signature::nullary in 44.0.0 easier / less confusing #13763 / Improve error messages for incorrect zero argument signatures #13881
• Make DocBuilder migration in 44.0.0 easier #13764 / Restore DocBuilder::new() to avoid breaking API change #13870

Pre-release testing:

• Comet upgrade looks good, thanks to @andygrove: chore: Upgrade to DataFusion 44.0.0-rc2 datafusion-comet#1154 (comment)
• Delta-rs upgrade looks good, thanks to @rtyler and @ion-elgreco: chore: api changes for Datafusion 44 delta-io/delta-rs#3073
• datafusion-python looks good, thanks to @timsaucer : Chore/upgrade datafusion 44 datafusion-python#973

Other items:

• Needs triage (maybe is already fixed): Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510

StringView support

These are mostly waiting on arrow-rs updates (I think we can fix many of them with the arrow-rs 53.4.0 release

• [Epic] A Collection of Additional UTF8View support tickets #13504

[alamb]

I am looking at #13510 now, and then plan to make a PR with release notes, verison update, etc

Update:

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510 has a workaround and is no worse than in 43 (though it is a regression since 42) so in my opinion this is not a blocker for the 44 release (it would be nice to fix though)

[andygrove]

@alamb The miri issue does not seem to be resolved for Comet - #13835 (comment)

Is there anything I should have done other than set default-features=false on all DF crates?

[alamb]

@alamb The miri issue does not seem to be resolved for Comet - #13835 (comment)

Is there anything I should have done other than set default-features=false on all DF crates?

That sounds like it is the right thing to me 🤔

• Since @buraksenn did the work to make this optional in [minor] make recursive package dependency optional  #13778, perhaps they have some idea?

[alamb]

Since @buraksenn did the work to make this optional in [minor] make recursive package dependency optional #13778, perhaps they have some idea?

Update I think I know what is wrong -- working on a fix now - #13887

[alamb]

I think all we are waiting on now is someone to review / approve (or fix in some other way) the recrusive-protection feature fix:

• Fix recursive-protection feature flag #13887

[alamb]

Update, when preparing to make a final RC I found another issue:

• swap_hash_join is no longer public so comet doesn't compile #13898

I have a proposed fix here:

• Fix visibility of swap_hash_join to be pub  #13899

[alamb]

Ok, we are clear for 🚀 from what I can tell -- the comet upgrade is working. I will merge the following PR

• Prepare for 44.0.0 release: version and changelog #13882 (thanks @xudong963 and @phillipleblanc for the review 🙏 )
  And then make a Release Candidate

[alamb]

I made a release candidate: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

[shehabgamin]

I made a release candidate: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

I'll re-test Sail with the release candidate tonight!

[shehabgamin]

Test 2 complete! Smooth Sailing testing commit 073a3b1 from DataFusion Release Candidate branch.

Test details here: #13855 (comment)

[alamb]

Thanks @shehabgamin

[timsaucer]

If we can make a new RC I would recommend cherry picking this fix for initcap: #13909

[alamb]

As @timsaucer mentions above (and on the mailing list): https://lists.apache.org/thread/34xy1nm47tkrr8lszxzwcrlf5b47h333

There was a regression introduced into initcap that has been sinced fixed: #13909

I will make a new RC

[alamb]

have created a new RC and started a new vote thread: https://lists.apache.org/thread/p0hyr78zs03f3snj0mh3vqm6c3kjwshn