EPIC: Improve Server-side Python experience

[pete-petey]

Work on client API inspired a better way to approach the server-side implementation.

@jmao-denver Please make tickets and link to this one.

• Basic plumbing for revamping Python integration #1294
• Hand wrap the new CSV implementation in Python #1391
• Wrap the Table interface in Python #1360
• Hand-wrap the Table class in Python #1486
• A barebone csv.write method in the new server Python package #1542
• [] Complete the reorg of TableTools.py for the new Python package #1553
• [] Add aggFormula support in deephaven2 #1699
• [] Remake conversion_utils.py and change NULL_CHAR to MAX_VALUE #1546
• Python not matching by aggregation with lists #1101
• Python bad matching by with list #1102
• [] Wrap the new Filter class to support the new where methods #1613
• [] Hand craft a DateTimeUtils wrapper #1640
• [] hand-written wrappers for Kafka/Parquet tools #1643

[jmao-denver]

The stated goal of the epic is to make the server side Python integration (the deephaven package) much more Pythonic, thus providing a better experience to the Python developers. A parallel goal is to make it as seamless as possible for a client side Python script to run on the server and vice versa. The latter requires we have an identical or very similar design of the Python API on both the server and the client.

After having carefully examining the current implementation of Python integration on the server side, I have come to realize that the scope of the work is quite a bit bigger than I first thought. These are the main reasons behind that realization. Some of these reasons especially 1, 2, also argue strongly against a partial effort.

• io.deephaven.db.tables.Table is exposed at run-time via JPY, to wrap it in a hand-written Python class means that we need to wrap all the methods/functions that either return a Table object or receive one as an input. Otherwise, we will have two forms of ‘Table’ classes in the package, which is a nightmare none would want.
• The code style and naming conventions of the current Python package are of Java’s and are not consistent with that of the new Python client which follows the Python styles and conventions. It’d be best that we just wrap up all the Java classes that need to be accessible from Python and make them all consistent in terms of code style and naming convention.
• The documentation of the current Python package leaves a lot to be desired, as it is mostly generated from Java Doc, definitely not Pythonic with overloads and Java FQN all over the places. To make the documentation more Pythonic, we need to hand-craft proper Python docstrings and will probably have to wrap up a bit more Java classes than anticipated.
• A properly written Python wrapper of a Java class will also need to take care of data type mappings between Python and Java beyond what JPY offers.
• A JPY runtime exposed class makes avaiable all its public methods and members, a wrapper of it might need to do some curation work on those.
• The final hand-written Python package should also have a consistent Pythonic error handling strategy, this is something the current one has overlooked.
• Technical debt in the current Python package: misplaced functions, not-so-coherent content structure, and other questionable decisions means we need to do some good reorganization.
• A more comprehensive test suite would be nice for the new Python package. The current one can afford having minimal test cases because its directly-mapped nature and that the mapped engine Java classes should have adequate test coverage already. The new can still enjoy the same benefit but with additional Python code, it'd be best that we have a proper test suite.
• Current DHCC documentation/examples need to be checked against the rewritten package and updated accordingly.
• My lack of experience with some of the features in the current package beyond table operations will be a factor in how fast the project can move.

[jmao-denver]

Deephaven V2 went out in the March release. This EPIC is considered done.