Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Remove cudf Cython bindings (cudf._lib) in favor of pylibcudf #17317

Open
4 tasks
mroeschke opened this issue Nov 14, 2024 · 0 comments
Open
4 tasks

[FEA] Remove cudf Cython bindings (cudf._lib) in favor of pylibcudf #17317

mroeschke opened this issue Nov 14, 2024 · 0 comments
Assignees
@mroeschke
Labels
feature request New feature or request Python Affects Python cuDF API.

Comments

@mroeschke
Copy link
Contributor

mroeschke commented Nov 14, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.
With pylibcudf largely complete in containing the necessary APIs for cudf Python (xref #15162), the cudf Cython layer (cudf._lib) should be largely unneeded now as pylibcudf provides a public interface to access libcudf.

Describe the solution you'd like
Ideally to fully remove the cudf._lib directory and stop developing Cython binding for this directory.

Each Cython file in cudf._lib should be able to be converted to a Python in cudf.core._internals (or cudf.core._plc?)

Additional context

Open Questions/Considerations

  1. cudf Cython spill locks whenever calling pylibcudf algorithms that operate on columns (e.g. nans_to_nulls), while there's currently no way to replicate spill locking with using pylibcudf alone.

  2. cudf Cython also defines Cython classes that back cudf Python columns and scalars. Ideally cudf Python should just be able to define these objects in Python, but it's not clear yet (to me) if there's some aspect that needs to be defined in Cython.

Projects using cudf Cython layer
@mroeschke mroeschke added the feature request New feature or request label Nov 14, 2024
@vyasr vyasr added the Python Affects Python cuDF API. label Nov 14, 2024
@mroeschke mroeschke self-assigned this Nov 14, 2024
This was referenced Nov 14, 2024
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 15, 2024
@mroeschke
Remove cudf._lib.avro in favor of inlining pylicudf (#17319)
d67d017 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17319
This was referenced Nov 15, 2024
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Move cudf._lib.unary to cudf.core._internals (#17318)
aeb6a30 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17318
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.hash in favor of inlining pylibcudf (#17345)
18b40dc 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17345
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.concat in favor of inlining pylibcudf (#17344)
ba21673 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17344
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.quantiles in favor of inlining pylibcudf (#17347)
02c35bf 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17347
rapids-bot bot pushed a commit that referenced this issue Nov 18, 2024
@mroeschke
Remove cudf._lib.labeling in favor of inlining pylibcudf (#17346)
302e625 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17346
This was referenced Nov 19, 2024
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 22, 2024
@mroeschke
Remove cudf._lib.transpose in favor of inlining pylibcudf (#17365)
881afd1 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #17365
@mroeschke mroeschke mentioned this issue Nov 22, 2024
Merged
3 tasks
@mroeschke mroeschke mentioned this issue Dec 4, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Dec 5, 2024
@mroeschke
Remove cudf._lib.merge in favor of inlining pylibcudf (#17370)
06e937b 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17370
rapids-bot bot pushed a commit that referenced this issue Dec 6, 2024
@mroeschke
Move cudf._lib.aggregation to cudf.core._internals (#17516)
c0a4c6c 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17516
@mroeschke mroeschke mentioned this issue Dec 6, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Dec 6, 2024
@mroeschke
Remove cudf._lib.text in favor of inlining pylibcudf (#17408)
c791f80 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17408
rapids-bot bot pushed a commit that referenced this issue Dec 6, 2024
@mroeschke
Remove cudf._lib.round in favor of inlining pylibcudf (#17430)
1a62b46 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17430
rapids-bot bot pushed a commit that referenced this issue Dec 6, 2024
@mroeschke
Remove cudf._lib.orc in favor of inlining pylibcudf (#17466)
b6f7e6e 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #17466
This was referenced Dec 7, 2024
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Dec 9, 2024
@mroeschke
Remove cudf._lib.string.convert/split in favor of inlining pylibcudf (#…
f595592 
…17496)

Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17496
rapids-bot bot pushed a commit that referenced this issue Dec 9, 2024
@mroeschke
Remove cudf._lib.transform in favor of inlining pylibcudf (#17505)
9df95d1 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17505
This was referenced Dec 10, 2024
Merged
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Dec 12, 2024
@mroeschke
Remove cudf._lib.parquet in favor of inlining pylibcudf (#17562)
92652be 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17562
@mroeschke mroeschke mentioned this issue Dec 12, 2024
Merged
3 tasks
rapids-bot bot pushed a commit that referenced this issue Dec 13, 2024
@mroeschke
Remove cudf._lib.nvtext in favor of inlining pylibcudf (#17535)
8a3e5f1 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17535
rapids-bot bot pushed a commit that referenced this issue Dec 13, 2024
@mroeschke
Remove cudf._lib.csv in favor in inlining pylibcudf (#17485)
7749702 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17485
rapids-bot bot pushed a commit that referenced this issue Dec 13, 2024
@mroeschke
Remove cudf._lib.reduce in favor of inlining pylibcudf (#17574)
48aa08f 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17574
rapids-bot bot pushed a commit that referenced this issue Dec 13, 2024
@mroeschke
Move cudf._lib.sort to cudf.core._internals (#17488)
1a67646 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17488
rapids-bot bot pushed a commit that referenced this issue Dec 16, 2024
@mroeschke
Remove unused functionality in cudf._lib.utils.pyx (#17586)
469f226 
Contributes to #17317

More can be removed once my other cudf._lib PRs are in

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17586
rapids-bot bot pushed a commit that referenced this issue Dec 16, 2024
@mroeschke
Move cudf._lib.copying to cudf.core._internals (#17548)
e975ca3 
Contributes to #17317

Also I found that `PackedColumns` was not being use anywhere. It appears it was added back in #8153 for dask_cudf but I cannot see it being used there anymore

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17548
rapids-bot bot pushed a commit that referenced this issue Dec 17, 2024
@mroeschke
Move cudf._lib.stream_compaction to cudf.core._internals (#17456)
c650bf7 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17456
rapids-bot bot pushed a commit that referenced this issue Dec 17, 2024
@mroeschke
Remove cudf._lib.string_casting in favor of inlining pylibcudf (#17460)
187053a 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17460
rapids-bot bot pushed a commit that referenced this issue Dec 17, 2024
@mroeschke
Remove cudf._lib.interop in favor of inlining pylibcudf (#17555)
b9760ac 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17555
This was referenced Dec 18, 2024
Merged
Merged
rapids-bot bot pushed a commit that referenced this issue Dec 19, 2024
@mroeschke
Remove cudf._lib.groupby in favor of inlining pylibcudf (#17582)
989fac4 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17582
rapids-bot bot pushed a commit that referenced this issue Dec 20, 2024
@mroeschke
Remove cudf._lib.utils in favor of python APIs (#17625)
69d62cb 
Contributes to #17317

Dependent on #17582

Did a search across RAPIDS and Morpheus and didn't find usage of these methods.

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17625
rapids-bot bot pushed a commit that referenced this issue Dec 20, 2024
@mroeschke
Use more pylibcudf.types instead of cudf._lib.types (#17619)
b06f510 
Contributes to #17317

Primary change is to use `pylibcudf.TypeId` instead of an ad-hoc one defined in `cudf._lib.types`. Additionally uses pylibcudf more consistently and inlines/removes some seldom uses/dead code

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17619
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mroeschke mroeschke

Labels
feature request New feature or request Python Affects Python cuDF API.
Projects
cuDF Python
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vyasr @mroeschke