Fugue Integration

[goodwanghan]

Description

This repository needs to integrate with Fugue so whyllogs profiling can work on Spark, Dask and Ray agnostically.

The commits in this pull request will:

1. Finish the initial implementation of profiling with/without logical partitioning
2. Docs and instructions

• I have reviewed the Guidelines for Contributing and the Code of Conduct.

[TheMellyBee]

I have a couple of questions and some comments around cleaning it up and clarity. Over all though it's looking in a good direction.

[jamie256]

This approach looks really nice to integrate whylogs and fugue! Let me know how I can help.

This looks close, I think with some minor cleanup if you have your example notebook working, and the CI checks passing we can merge this? I will check tomorrow morning, and happy to help resolve issues.

[goodwanghan]

Thank @jamie256 the notebook is working. I will also add some comments on the public function. Then we are good to merge. Currently I am having a hard time with poetry I am hitting this issue python-poetry/poetry#4242 and may try poetry 1.2 which fixes it.

[jamie256]

I am hitting this issue python-poetry/poetry#4242 and may try poetry 1.2 which fixes it.

I updated poetry to version 1.2.1 in our CI configs as it now works after fixing the typing-extensions entry: #847

[goodwanghan]

I am hitting this issue python-poetry/poetry#4242 and may try poetry 1.2 which fixes it.

I updated poetry to version 1.2.1 in our CI configs as it now works after fixing the typing-extensions entry: #847

Thanks, but please let me do that, I am making a couple of changes

[goodwanghan]

Merging poetry.lock is not fun 😢

[jamie256]

Merging poetry.lock is not fun 😢

Looks like you have it working! And notebooks cleaned up.

poetry.lock should be generated from pyproject.toml file changes by running poetry update (if possible), manually updating the lock file is painful.

[jamie256]

Thanks for the push and all the updates here, I'll pull this into today's release.

[andyndang]

lgtm!