String dtype: implement sum reduction

[jorisvandenbossche]

Based on the feedback in #59328, implementing sum() for the string dtype

xref #54792

[rhshadrach]

Not too familiar here, can this cause unexpected results? If so, should it be documented?

[jorisvandenbossche]

Yes, it can cause overflow error if a single chunk doesn't fit into the string dtype. I suppose this will be very rare, because we are summing here, and that would mean that the single scalar string as result of the sum would be bigger than 2GB (I am not fully sure how well Python will handle such a large str object).

We can indeed document it.
In theory it could also be circumvented by splitting the chunk into multiple chunks (although I have to verify that pyarrow then does not actually concatenate that again under the hood in the binary_join implementation).

[jorisvandenbossche]

Added a note about this in the issue listing the behavioral changes -> #59328

[WillAyd]

Hmm wouldn't it be better to just make users cast to object if they want the legacy behavior on this? I think it was intentional to not implement sum on the string dtype; while there may be some niche use cases, more often than not I think its a mistake (and huge performance hit) to have string types summed up

[rhshadrach]

while there may be some niche use cases, more often than not I think its a mistake (and huge performance hit)

This is not my personal experience.

[jorisvandenbossche]

@WillAyd it was indeed intentional in the past to not implement it, but see the thread in #59328, and if you want to discuss this, let's do that there ;) (I personally don't care too much about this specifically, but can follow some of the arguments given there, and just implementing what has been decided for now)

[jorisvandenbossche]

I am planning to merge this tomorrow (given it is one of the last remaining bigger changes for the string dtype, and including a bunch of test updates / blocking other test updates), but some more review is certainly still welcome

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.3.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 2fdb16b347fc34f78213868a8a973447ac79ab2d

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #59853: String dtype: implement sum reduction'

4. Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-59853-on-2.3.x

5. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #59853 on branch 2.3.x (String dtype: implement sum reduction)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[jorisvandenbossche]

Manual backport -> #60157