Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

port the numba integration prototype from Enterprise to OSS, fixes feature-523 #638

Merged
merged 3 commits into from
May 24, 2021

Conversation

jmao-denver
Copy link
Contributor

this prototype in its current design must be used in an unpublished, explicit way. So it should not break anything visible to users.

@jmao-denver jmao-denver added this to the May 2021 milestone May 19, 2021
@jmao-denver jmao-denver requested a review from devinrsmith May 19, 2021 20:15
@jmao-denver jmao-denver self-assigned this May 19, 2021
Copy link
Member

@devinrsmith devinrsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, mostly. Couple changes needed. Hard for me to tell if there are any big departures from how I originally structured my patch?

I'll be testing this out locally now.

py/jpy/src/main/java/org/jpy/PyObject.java Outdated Show resolved Hide resolved
@devinrsmith
Copy link
Member

Great - was able to test the following, and it works:

from deephaven import TableTools, java, numba as dnb

@dnb.vectorize
def my_manual(x: java.int, y: java.int) -> java.int:
  return (x % 3) + y

IJ = TableTools.emptyTable(1000000) \
    .view("I=(int)i", "J=(int)(i * 2)") \
    .update(java.formula(my_manual, name = "K", column_names = ["I", "J"]))

@devinrsmith
Copy link
Member

devinrsmith commented May 20, 2021

Good example of the chunking working:

from deephaven import TableTools, java, numba as dnb

@dnb.vectorize
def my_vectorized(x: java.int, y: java.int) -> java.int:
  return (x % 3) + y

vectorized_sum = TableTools.emptyTable(100000000) \
    .view("I=(int)i", "J=(int)(i * 2)") \
    .view(java.formula(my_manual, name = "K", column_names = ["I", "J"])) \
    .sumBy()

is very fast

vs

from deephaven import TableTools, java

def my_plain(x: java.int, y: java.int) -> java.int:
  return (x % 3) + y
  
plain_sum = TableTools.emptyTable(100000000) \
    .view("I=(int)i", "J=(int)(i * 2)") \
    .view("K = (int)my_plain(I, J)") \
    .sumBy()

is very slow.

@jmao-denver jmao-denver requested a review from devinrsmith May 24, 2021 14:29
@jmao-denver jmao-denver merged commit 5c1da97 into deephaven:main May 24, 2021
devinrsmith added a commit to devinrsmith/deephaven-core that referenced this pull request May 25, 2021
…-check

* upstream/main:
  Check documentation labels, fixes deephaven#627 (deephaven#671)
  Fix integer overflow in SortedRanges.invertOnNew.  Fixes deephaven#666. (deephaven#667)
  Fix integer overflow in SortedRanges.subRangeByPos. Fixes deephaven#664 (deephaven#665)
  Deploy example scripts and data (deephaven#562)
  port the numba integration prototype from Enterprise to OSS, fixes feature-523 (deephaven#638)
  Fix IllegalStateException in TwoValuesContainer triggered from Index.insert(Chunk). Fixes deephaven#652. (deephaven#653)
  Javadoc search (deephaven#645)
  Support for jpy integration junit tests (deephaven#632)
  Populate snapshot request in js api, handle optional left table (deephaven#648)
  Fix a link formatting bug in CONTRIBUTING.md (deephaven#629)
devinrsmith added a commit to devinrsmith/deephaven-core that referenced this pull request May 28, 2021
* tag 'v0.0.2': (29 commits)
  Move label information to CI
  remove labels
  try pushing to my own org
  try actor
  Upgrade to GITHUB_TOKEN instead of PAT
  Explicit about docker/build-push-action version
  Explicit about docker/login-action version
  Explicit about docker/setup-buildx-action version
  Remove extraneous comments
  Update docker metadata steps
  Remove top level building of deephaven-jpy in CI, is handled implicitly in grpc-api step
  Fix nightly check (deephaven#641)
  Address `QueryTableTest.testReverseClipping` table reachability bug, and unit test listeners using `RetentionCache` (deephaven#644)
  Rename symbols in .proto files to follow standard conventions (deephaven#674)
  Check documentation labels, fixes deephaven#627 (deephaven#671)
  Fix integer overflow in SortedRanges.invertOnNew.  Fixes deephaven#666. (deephaven#667)
  Fix integer overflow in SortedRanges.subRangeByPos. Fixes deephaven#664 (deephaven#665)
  Deploy example scripts and data (deephaven#562)
  port the numba integration prototype from Enterprise to OSS, fixes feature-523 (deephaven#638)
  Fix IllegalStateException in TwoValuesContainer triggered from Index.insert(Chunk). Fixes deephaven#652. (deephaven#653)
  ...
jmao-denver added a commit to jmao-denver/deephaven-core that referenced this pull request Apr 11, 2022
…ature-523 (deephaven#638)

* port the numba integration prototype from Enterprise to OSS, WIP, fixes feature-523

* changes to make python unittest happy

* restore important javadoc comments to the <type>Chunk.java files, fixes deephaven#523
@jmao-denver jmao-denver deleted the feature-523 branch February 8, 2023 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants