Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hand wrapped KafkaTools, ParquetTools, fixes #1643, #1546, #1892 #1825

Merged
merged 15 commits into from
Feb 16, 2022
Merged

Hand wrapped KafkaTools, ParquetTools, fixes #1643, #1546, #1892 #1825

merged 15 commits into from
Feb 16, 2022

Conversation

jmao-denver
Copy link
Contributor

  1. created stream/kafka package which included the consumer.py and producer.py modules
  2. added the parquet.py. module
  3. converted and added docstrings to the new standards
  4. refitted the existing test code to the new interfaces and added some new cases

@jmao-denver jmao-denver added this to the Jan 2022 milestone Jan 13, 2022
@jmao-denver jmao-denver self-assigned this Jan 13, 2022
@jmao-denver jmao-denver changed the title Hand wrapped KafkaTools, ParquetTools, fixes #1643 Hand wrapped KafkaTools, ParquetTools, fixes #1643, #1546 Jan 13, 2022
jcferretti
jcferretti reviewed Jan 14, 2022
View reviewed changes
Copy link
Member

@jcferretti jcferretti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed Kafka and Parquet parts of the diff; can't comment to changes to pre-existing files in pyintegration/deephaven2

pyintegration/deephaven2/stream/kafka/consumer.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/_utils.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/__init__.py Outdated Show resolved Hide resolved
chipkent
chipkent requested changes Jan 26, 2022
View reviewed changes
pyintegration/deephaven2/__init__.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/__init__.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/_init/bootstrap.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/_init/bootstrap.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/tests/test_parquet.py Outdated Show resolved Hide resolved
pyintegration/tests/test_parquet.py Show resolved Hide resolved
pyintegration/tests/test_parquet.py Show resolved Hide resolved
pyintegration/tests/test_parquet.py Outdated Show resolved Hide resolved
pyintegration/tests/test_produce_kafka.py Outdated Show resolved Hide resolved
@jmao-denver jmao-denver force-pushed the feature-1643 branch 2 times, most recently from 64854fa to eef89e3 Compare January 31, 2022 18:08
@jmao-denver
Hand wrapped KafkaTools, ParquetTools
26443f3
@jmao-denver
Fixed a test failure caused by testing env change
5d2e7fd 
Fixed some pydocs issues, typehints
Updated kafka/producer to reflect recent update to the Java API
@jmao-denver
Refactored away poorly named utils modules
1c088fe 
1. Turned functions that create instances of Java util types(HashMap
/HashSet/Properties/ArrayList) into DType callables in dtypes module
2. Removed the utils modules used to host the above functions
@jmao-denver
Made config moduel more useful
fbe90d8 
Also add round-trip check in Parquet tests
@jmao-denver jmao-denver changed the title Hand wrapped KafkaTools, ParquetTools, fixes #1643, #1546 Hand wrapped KafkaTools, ParquetTools, fixes #1643, #1546, #1892 Feb 3, 2022
@jmao-denver
Simplified both parquet.py and kafka consumer.py
f3d89f9 
also cleaned up the docstrings and added a generic wrapper ABC class
@jmao-denver jmao-denver requested a review from chipkent February 4, 2022 02:29
@jcferretti jcferretti mentioned this pull request Feb 4, 2022
Closed
chipkent
chipkent requested changes Feb 11, 2022
View reviewed changes
pyintegration/deephaven2/config/__init__.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/parquet.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/parquet.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/consumer.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/consumer.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/table.py Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/producer.py Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/producer.py Outdated Show resolved Hide resolved
pyintegration/tests/test_parquet.py Show resolved Hide resolved
@jmao-denver
Add missing docstrings and simplified dtypes
dabb3f7
@jmao-denver
Simplify dtypes module
3b771a0 
Refactored DType to move out functionalities that don't belong to
make it more cohesive
chipkent
chipkent requested changes Feb 14, 2022
View reviewed changes
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/dtypes.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/parquet.py Outdated Show resolved Hide resolved
@jmao-denver
Create the private java compatibility module
82d75c0 
More pydoc addtion and cleanup
More dtypes code cleanup
chipkent
chipkent requested changes Feb 15, 2022
View reviewed changes
pyintegration/deephaven2/_jcompat.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/producer.py Outdated Show resolved Hide resolved
pyintegration/deephaven2/stream/kafka/producer.py Outdated Show resolved Hide resolved
@deephaven deephaven deleted a comment from chipkent Feb 15, 2022
@jmao-denver jmao-denver requested a review from chipkent February 16, 2022 03:04
@chipkent chipkent modified the milestones: Jan 2022, Feb 2022 Feb 16, 2022
@jmao-denver jmao-denver merged commit 003f272 into deephaven:main Feb 16, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Feb 16, 2022
@jmao-denver jmao-denver deleted the feature-1643 branch February 8, 2023 18:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jcferretti jcferretti jcferretti left review comments

@cpwright cpwright cpwright left review comments

@devinrsmith devinrsmith devinrsmith left review comments

@rcaudy rcaudy rcaudy left review comments

@chipkent chipkent chipkent approved these changes

Assignees

@jmao-denver jmao-denver

Labels
NoDocumentationNeeded python
Projects
None yet
Milestone
Feb 2022
Development

Successfully merging this pull request may close these issues.
6 participants
@jmao-denver @chipkent @rcaudy @devinrsmith @cpwright @jcferretti