Add Python Dependency Deck #2264

jasonlai1218 · 2024-03-13T17:32:44Z

Tracking issue

Why are the changes needed?

In Python Dependency Deck, the version of the Python package currently being used can be displayed, which can make debugging easier.

What changes were proposed in this pull request?

Add a class PythonDependencyDeck to deck.py
Add a method add_deck to FlyteContext class

How was this patch tested?

Setup process

import flytekit
from flytekit import task, workflow, ImageSpec

flytekit_version = "git+https://github.com/jasonlai1218/flytekit.git@add-py-deps-deck#commit=b5f49b4070621ba87224c9d223a6f7d710084673"
deck_plugin_version = "git+https://github.com/jasonlai1218/flytekit.git@add-py-deps-deck#commit=b5f49b4070621ba87224c9d223a6f7d710084673#subdirectory=plugins/flytekit-deck-standard"

image_spec = ImageSpec(
    packages=[flytekit_version, deck_plugin_version],
    apt_packages=["git"],
    registry="jasonlai1218",
    env={"FLYTE_SDK_LOGGING_LEVEL": "20"},
)


@task(container_image=image_spec, enable_deck=True)
def python＿dependency_deck() -> None:
    ctx = flytekit.FlyteContextManager.current_context()
    ctx.add_deck(PythonDependencyDeck())


@workflow
def wf() -> None:
    python＿dependency_deck()


if __name__ == "__main__":
    wf()

Screenshots

click copy button and paste result:

adlfs==2024.2.0
aiobotocore==2.12.1
aiohttp==3.9.3
aioitertools==0.11.0
aiosignal==1.3.1
annotated-types==0.6.0
arrow==1.3.0
asttokens==2.4.1
attrs==23.2.0
azure-core==1.30.1
azure-datalake-store==0.0.53
azure-identity==1.15.0
azure-storage-blob==12.19.1
binaryornot==0.4.4
botocore==1.34.51
...
...
...
websocket-client==1.7.0
widgetsnbextension==4.0.10
wordcloud==1.9.3
wrapt==1.16.0
yarl==1.9.4
ydata-profiling==4.7.0
zipp==3.18.1

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

flyteorg/flyteconsole#852

Docs link

- Add a new class `SourceCodeDeck` to render the source code of a task - Implement the `html` property in the `SourceCodeDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Change the class name from `SourceCodeDeck` to `PythonDependencyDeck` - Update the deck description to reflect python dependencies instead of source code - Remove the source code rendering functionality and replace it with a table renderer for python dependencies - Add subprocess logic to fetch installed python packages and render them in an HTML table Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add a property method for `pythondependency_deck` - Initialize `python_dependency_deck` if it is None - Remove a property method for `timeline_deck` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add `PythonDependencyDeck` to `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add `pandas` import to `flytekit/deck/deck.py` - Change the input of `TableRenderer().to_html` to a `pd.DataFrame` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Update the return type annotation for `pythondependency_deck` in `context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add `PythonDependencyDeck` to the TYPE_CHECKING import - Add a method `add_deck` to `FlyteContext` class Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Refactor the code to use a DataFrame for installed packages handling Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Update the name of the PythonDependencyDeck instance from "PythonDependencyDeck" to "Python Dependency" Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add CSS style to center align the table content Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Rename method `pythondependency_deck` to `python_dependency_deck` - Update variable names in method `python_dependency_deck` - Update method comments in class `FlyteContext` - Update method comments in class `PythonDependencyDeck` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Remove import of `PythonDependencyDeck` from `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Remove the import of `PythonDependencyDeck` from `flytekit/core/context_manager.py` - Add an import of `Deck` to `flytekit/core/context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Add a condition to check for deck existence before appending - Change single quotes to double quotes for consistency - Update package split delimiter to double quotes Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Remove the `python_dependency_deck` property from the `ExecutionParameters` class in `context_manager.py` - Update the `__init__` method signature in the `PythonDependencyDeck` class in `deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

codecov · 2024-03-13T18:06:31Z

Codecov Report

Attention: Patch coverage is 90.32258% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 82.68%. Comparing base (55f0b19) to head (814a952).

Files	Patch %	Lines
flytekit/deck/renderer.py	85.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2264      +/-   ##
==========================================
- Coverage   83.50%   82.68%   -0.83%     
==========================================
  Files         324      305      -19     
  Lines       24754    24063     -691     
  Branches     3734     3523     -211     
==========================================
- Hits        20672    19896     -776     
- Misses       3452     3542      +90     
+ Partials      630      625       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

- Remove unnecessary import of `pandas` in `flytekit/deck/deck.py` - Add import of `pandas` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

flytekit/deck/deck.py

thomasjpfan

Thank you for the PR!

flytekit/deck/deck.py

- Add a new test for the PythonDependencyDeck class in test_deck.py - Add assertions for specific strings in the HTML content in the test_python_dependency_deck() function Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Clear user space decks before adding a new deck - Ensure only one deck is added to user space params Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Import json and TableRenderer in `flytekit/deck/deck.py` - Change how installed packages are fetched in `flytekit/deck/deck.py` - Remove unused code in `flytekit/deck/deck.py` - Remove assertions for "Library" and "Version" in `tests/flytekit/unit/deck/test_deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

flytekit/deck/deck.py

- Modify the table headers to have left-aligned text - Add extra line breaks after the button element Signed-off-by: jason.lai <jasonlai1218@gmail.com>

tests/flytekit/unit/deck/test_deck.py

- Change the assertion to check for `Name` and `Version` instead of `name` and `version` in the result Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- Increase the test deadlines from 2 to 20 seconds in test_type_conversion_errors.py - Update test deadlines from 2 to 40 seconds in test_type_conversion_errors.py - Adjust test deadlines in test_eager_workflows.py for various tests Signed-off-by: jason.lai <jasonlai1218@gmail.com>

This reverts commit 1885662. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

…ow that we're no longer writing decks unnecessarily) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

- Add a patch to the `test_python_dependency_renderer` function - Include assertions for `numpy` and `1.21.0` in the test result - Update the `test_deck.py` file in the `tests/flytekit/unit/deck` directory Signed-off-by: jason.lai <jasonlai1218@gmail.com>

flytekit/core/python_function_task.py

flytekit/deck/renderer.py

- Import `PythonDependencyRenderer` and `SourceCodeRenderer` in `PythonFunctionTask` from separate files - Modify the way `requirements_txt` is generated in `PythonDependencyRenderer` - Refactor the creation of the `table` variable in `PythonDependencyRenderer` to improve readability Signed-off-by: jason.lai <jasonlai1218@gmail.com>

thomasjpfan · 2024-04-02T19:17:12Z

flytekit/deck/renderer.py

+            )
+        except subprocess.CalledProcessError as e:
+            logger.error(f"Error occurred while fetching installed packages: {e}")
+            return ""


Having an empty HTML feels like a bad dev experience. I think it's worth having an error message:

Suggested change

return ""

return "Error occurred while fetching installed packages."

thomasjpfan · 2024-04-02T19:18:42Z

flytekit/deck/renderer.py

+        </head>
+        <body>
+
+        <button onclick="copyTable()">


@eapolinario This copy button does not work. I get this error in the console:

If we can not get this to work, I'm okay with removing the copy button and just showing the table.

This is blocked on flyteorg/flyteconsole#852.

That just got merged, so next Flyte release is going to have this.

thomasjpfan · 2024-04-02T19:22:36Z

flytekit/core/python_function_task.py

-            source_code_deck = Deck("Source Code")
-            renderer = SourceCodeRenderer()
-            source_code_deck.append(renderer.to_html(source_code))
+            python_dependencies_deck = Deck("Python Dependencies")


Nit: Making this shorter looks better in the UI:

Suggested change

python_dependencies_deck = Deck("Python Dependencies")

python_dependencies_deck = Deck("Dependencies")

"Python Dependencies using a second line":

thomasjpfan · 2024-04-02T19:23:54Z

flytekit/deck/renderer.py

+    PythonDependencyDeck is a deck that contains information about packages installed via pip.
+    """
+
+    def __init__(self, title: str = "Python Dependencies"):


Suggested change

def __init__(self, title: str = "Python Dependencies"):

def __init__(self, title: str = "Dependencies"):

thomasjpfan · 2024-04-02T19:25:46Z

flytekit/deck/renderer.py

+        <button onclick="copyTable()">
+          <span>Copy table as requirements.txt</span>
+        </button>
+        <br><br><br>


Suggested change

<br><br><br>

<h3>Python Dependencies</h3>

- Change the title of the deck from `Python Dependencies` to `Dependencies` - Update the error message in case of a subprocess error in fetching installed packages - Add a heading `Python Dependencies` above the table output in the HTML Signed-off-by: jason.lai <jasonlai1218@gmail.com>

thomasjpfan

Assuming https://github.com/flyteorg/flytekit/pull/2264/files#r1548469361 will get resolved with a new flyte release, this PR LGTM

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

eapolinario · 2024-04-03T18:18:38Z

FYI: I pushed a commit to handle tests that require hypothesis slightly differently. Now we run them via a separate make target (called unit_test_hypothesis) where we control some of the hyperparameters like deadline and max_examples using a profile (as per the docs).

* feat: refactor source code rendering in `SourceCodeDeck` class - Add a new class `SourceCodeDeck` to render the source code of a task - Implement the `html` property in the `SourceCodeDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor deck rendering for Python dependencies - Change the class name from `SourceCodeDeck` to `PythonDependencyDeck` - Update the deck description to reflect python dependencies instead of source code - Remove the source code rendering functionality and replace it with a table renderer for python dependencies - Add subprocess logic to fetch installed python packages and render them in an HTML table Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor class properties and initialization - Add a property method for `pythondependency_deck` - Initialize `python_dependency_deck` if it is None - Remove a property method for `timeline_deck` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * - feat: consolidate Python dependencies in `flytekit/__init__.py` - Add `PythonDependencyDeck` to `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor `flytekit/deck/deck.py` for `pandas` compatibility - Add `pandas` import to `flytekit/deck/deck.py` - Change the input of `TableRenderer().to_html` to a `pd.DataFrame` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor return types across multiple files - Update the return type annotation for `pythondependency_deck` in `context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: consolidate Python dependency management in FlyteContext - Add `PythonDependencyDeck` to the TYPE_CHECKING import - Add a method `add_deck` to `FlyteContext` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor code to use DataFrame for package handling - Refactor the code to use a DataFrame for installed packages handling Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: refactor code for improved naming conventions - Update the name of the PythonDependencyDeck instance from "PythonDependencyDeck" to "Python Dependency" Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve table alignment styling in CSS - Add CSS style to center align the table content Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor method and variable names across files - Rename method `pythondependency_deck` to `python_dependency_deck` - Update variable names in method `python_dependency_deck` - Update method comments in class `FlyteContext` - Update method comments in class `PythonDependencyDeck` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor imports in flytekit package - Remove import of `PythonDependencyDeck` from `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: consolidate import statements in core/context_manager.py - Remove the import of `PythonDependencyDeck` from `flytekit/core/context_manager.py` - Add an import of `Deck` to `flytekit/core/context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve code consistency and error checking - Add a condition to check for deck existence before appending - Change single quotes to double quotes for consistency - Update package split delimiter to double quotes Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor Python dependency handling in classes - Remove the `python_dependency_deck` property from the `ExecutionParameters` class in `context_manager.py` - Update the `__init__` method signature in the `PythonDependencyDeck` class in `deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: optimize imports in deck.py files - Remove unnecessary import of `pandas` in `flytekit/deck/deck.py` - Add import of `pandas` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: improve test coverage for PythonDependencyDeck class - Add a new test for the PythonDependencyDeck class in test_deck.py - Add assertions for specific strings in the HTML content in the test_python_dependency_deck() function Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: enhance user space deck management - Clear user space decks before adding a new deck - Ensure only one deck is added to user space params Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor deck module and unit tests - Import json and TableRenderer in `flytekit/deck/deck.py` - Change how installed packages are fetched in `flytekit/deck/deck.py` - Remove unused code in `flytekit/deck/deck.py` - Remove assertions for "Library" and "Version" in `tests/flytekit/unit/deck/test_deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * fix: update subprocess calls to use `sys.executable` - Import `sys` to fix an issue with subprocess execution - Update the subprocess call to use `sys.executable` instead of `pip` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: refactor HTML generation logic and improve user experience - Update the HTML generation logic to include a button for copying the table as requirements.txt. Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: enhance table content copying functionality - Add functionality to copy table content as requirements.txt - Improve error handling when copying table content - Display table content as hidden div for copying Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: improve table content copying functionality - Remove unnecessary code for table content copying - Update table content copying functionality to use innerText of requirements_txt element Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: improve package management and error handling - Add logic to generate a `requirements.txt` file from installed packages - Update error logging message in case of subprocess error - Update console log message when accessing clipboard - Update `requirements_txt` div content with actual `requirements_txt` variable Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: standardize whitespace in requirements_txt handling - Remove trailing whitespace from `requirements_txt` string - Add a whitespace to the end of the `requirements_txt` variable - Log an error message when fetching installed packages fails Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize quotation marks for package_info keys - Corrected quotation marks in package_info keys - Changed single quotes to double quotes for consistency Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: simplify requirements_txt generation - Refactor code to simplify the generation of `requirements_txt` - Add the output of `pip freeze` to `requirements_txt` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: fix typos and improve code consistency across files - Fix a typo in the usage of the `pip freeze` command in the PythonDependencyDeck class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor dependency handling in PythonDependencyDeck class - Update the way `requirements_txt` is populated in the `PythonDependencyDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update default name and test assertion in PythonDependencyDeck - Update the default name in the PythonDependencyDeck constructor from "Python Dependency" to "Python Dependencies" - Update the assertion in the test_python_dependency_deck function to check for the new default name "Python Dependencies" Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update rendering of Pandas DataFrame using MarkdownRenderer - Import `MarkdownRenderer` from `flytekit.deck` instead of `TableRenderer` - Render the Pandas DataFrame as markdown using `MarkdownRenderer` instead of `TableRenderer` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor PythonDependencyDeck and related classes - Remove the `add_deck` method from `FlyteContext` - Add imports for `PythonDependencyRenderer` and `PythonDependencyDeck` in `PythonFunctionTask` - Remove the `PythonDependencyDeck` class and related methods from the `deck.py` file - Add the `PythonDependencyRenderer` class in `renderer.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: use `TableRenderer` for rendering DataFrames - Import `TableRenderer` from a different module - Replace the `MarkdownRenderer` with `TableRenderer` to render a DataFrame as a table Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: refactor codebase for improved performance - Remove unused import statements in `flytekit/deck/renderer.py` - Add a new test for `PythonDependencyRenderer` in `tests/flytekit/unit/deck/test_deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: update test_deck.py for python dependency deck testing - Update test_deck.py to include python dependency deck in various test cases - Adjust expected deck counts in test cases to reflect the changes - Add test cases for scenarios involving python dependency deck and input and output decks Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize import statements for pandas in project files - Remove `import pandas as pd` and replace it with `import pandas as pandas` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor import statements for `TableRenderer` usage - Import `TableRenderer` from a different location in `flytekit/deck/renderer.py` - Remove `TableRenderer` from the imports in `flytekit-deck-standard/flytekitplugins/deck/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update PythonDependencyRenderer description - Update the description of PythonDependencyDeck in PythonDependencyRenderer Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor type hints and assertions across files - Change the type hint for `df` parameter to `pandas.DataFrame` - Add an assertion to check the type of `df` is `pandas.DataFrame` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: refactor rendering classes in deck and plugins - Add TableRenderer to TimeLineDeck class in deck.py - Remove TableRenderer from deck/renderer.py - Add TableRenderer to PythonDependencyRenderer in deck/renderer.py - Add TableRenderer to flytekit-deck-standard plugin - Remove TableRenderer from test_renderer.py Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: standardize markdown formatting across files - Update the markdown table format in PythonDependencyRenderer class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor markdown table rendering in PythonDependencyRenderer - Update the markdown table format in the PythonDependencyRenderer class to use HTML table tags - Modify the markdown_table variable concatenation for installed packages entries in the PythonDependencyRenderer class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve code consistency in PythonDependencyRenderer - Add a line break after the button element in PythonDependencyRenderer Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize formatting for better readability - Modify the table headers to have left-aligned text - Add extra line breaks after the button element Signed-off-by: jason.lai <jasonlai1218@gmail.com> * fix: update assertion to check for `Name` and `Version` consistency - Change the assertion to check for `Name` and `Version` instead of `name` and `version` in the result Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: extend test deadlines across various tests - Increase the test deadlines from 2 to 20 seconds in test_type_conversion_errors.py - Update test deadlines from 2 to 40 seconds in test_type_conversion_errors.py - Adjust test deadlines in test_eager_workflows.py for various tests Signed-off-by: jason.lai <jasonlai1218@gmail.com> * Revert "test: extend test deadlines across various tests" This reverts commit 1885662. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Only run generate decks in python_function_task if decks are enabled Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove breakpoint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix test_deck.py tests to account for the number of expected decks (now that we're no longer writing decks unnecessarily) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Increase deadline of eager tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix lint error Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * test: update test assertions in test_deck.py - Add a patch to the `test_python_dependency_renderer` function - Include assertions for `numpy` and `1.21.0` in the test result - Update the `test_deck.py` file in the `tests/flytekit/unit/deck` directory Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor Python code organization - Import `PythonDependencyRenderer` and `SourceCodeRenderer` in `PythonFunctionTask` from separate files - Modify the way `requirements_txt` is generated in `PythonDependencyRenderer` - Refactor the creation of the `table` variable in `PythonDependencyRenderer` to improve readability Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: refactor project structure and improve user experience - Change the title of the deck from `Python Dependencies` to `Dependencies` - Update the error message in case of a subprocess error in fetching installed packages - Add a heading `Python Dependencies` above the table output in the HTML Signed-off-by: jason.lai <jasonlai1218@gmail.com> * Separate out tests that require hypothesis Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: jason.lai <jasonlai1218@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* feat: refactor source code rendering in `SourceCodeDeck` class - Add a new class `SourceCodeDeck` to render the source code of a task - Implement the `html` property in the `SourceCodeDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor deck rendering for Python dependencies - Change the class name from `SourceCodeDeck` to `PythonDependencyDeck` - Update the deck description to reflect python dependencies instead of source code - Remove the source code rendering functionality and replace it with a table renderer for python dependencies - Add subprocess logic to fetch installed python packages and render them in an HTML table Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor class properties and initialization - Add a property method for `pythondependency_deck` - Initialize `python_dependency_deck` if it is None - Remove a property method for `timeline_deck` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * - feat: consolidate Python dependencies in `flytekit/__init__.py` - Add `PythonDependencyDeck` to `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor `flytekit/deck/deck.py` for `pandas` compatibility - Add `pandas` import to `flytekit/deck/deck.py` - Change the input of `TableRenderer().to_html` to a `pd.DataFrame` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor return types across multiple files - Update the return type annotation for `pythondependency_deck` in `context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: consolidate Python dependency management in FlyteContext - Add `PythonDependencyDeck` to the TYPE_CHECKING import - Add a method `add_deck` to `FlyteContext` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor code to use DataFrame for package handling - Refactor the code to use a DataFrame for installed packages handling Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: refactor code for improved naming conventions - Update the name of the PythonDependencyDeck instance from "PythonDependencyDeck" to "Python Dependency" Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve table alignment styling in CSS - Add CSS style to center align the table content Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor method and variable names across files - Rename method `pythondependency_deck` to `python_dependency_deck` - Update variable names in method `python_dependency_deck` - Update method comments in class `FlyteContext` - Update method comments in class `PythonDependencyDeck` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor imports in flytekit package - Remove import of `PythonDependencyDeck` from `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: consolidate import statements in core/context_manager.py - Remove the import of `PythonDependencyDeck` from `flytekit/core/context_manager.py` - Add an import of `Deck` to `flytekit/core/context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve code consistency and error checking - Add a condition to check for deck existence before appending - Change single quotes to double quotes for consistency - Update package split delimiter to double quotes Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor Python dependency handling in classes - Remove the `python_dependency_deck` property from the `ExecutionParameters` class in `context_manager.py` - Update the `__init__` method signature in the `PythonDependencyDeck` class in `deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: optimize imports in deck.py files - Remove unnecessary import of `pandas` in `flytekit/deck/deck.py` - Add import of `pandas` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: improve test coverage for PythonDependencyDeck class - Add a new test for the PythonDependencyDeck class in test_deck.py - Add assertions for specific strings in the HTML content in the test_python_dependency_deck() function Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: enhance user space deck management - Clear user space decks before adding a new deck - Ensure only one deck is added to user space params Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor deck module and unit tests - Import json and TableRenderer in `flytekit/deck/deck.py` - Change how installed packages are fetched in `flytekit/deck/deck.py` - Remove unused code in `flytekit/deck/deck.py` - Remove assertions for "Library" and "Version" in `tests/flytekit/unit/deck/test_deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * fix: update subprocess calls to use `sys.executable` - Import `sys` to fix an issue with subprocess execution - Update the subprocess call to use `sys.executable` instead of `pip` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: refactor HTML generation logic and improve user experience - Update the HTML generation logic to include a button for copying the table as requirements.txt. Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: enhance table content copying functionality - Add functionality to copy table content as requirements.txt - Improve error handling when copying table content - Display table content as hidden div for copying Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: improve table content copying functionality - Remove unnecessary code for table content copying - Update table content copying functionality to use innerText of requirements_txt element Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: improve package management and error handling - Add logic to generate a `requirements.txt` file from installed packages - Update error logging message in case of subprocess error - Update console log message when accessing clipboard - Update `requirements_txt` div content with actual `requirements_txt` variable Signed-off-by: jason.lai <jasonlai1218@gmail.com> * chore: standardize whitespace in requirements_txt handling - Remove trailing whitespace from `requirements_txt` string - Add a whitespace to the end of the `requirements_txt` variable - Log an error message when fetching installed packages fails Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize quotation marks for package_info keys - Corrected quotation marks in package_info keys - Changed single quotes to double quotes for consistency Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: simplify requirements_txt generation - Refactor code to simplify the generation of `requirements_txt` - Add the output of `pip freeze` to `requirements_txt` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: fix typos and improve code consistency across files - Fix a typo in the usage of the `pip freeze` command in the PythonDependencyDeck class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor dependency handling in PythonDependencyDeck class - Update the way `requirements_txt` is populated in the `PythonDependencyDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update default name and test assertion in PythonDependencyDeck - Update the default name in the PythonDependencyDeck constructor from "Python Dependency" to "Python Dependencies" - Update the assertion in the test_python_dependency_deck function to check for the new default name "Python Dependencies" Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update rendering of Pandas DataFrame using MarkdownRenderer - Import `MarkdownRenderer` from `flytekit.deck` instead of `TableRenderer` - Render the Pandas DataFrame as markdown using `MarkdownRenderer` instead of `TableRenderer` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor PythonDependencyDeck and related classes - Remove the `add_deck` method from `FlyteContext` - Add imports for `PythonDependencyRenderer` and `PythonDependencyDeck` in `PythonFunctionTask` - Remove the `PythonDependencyDeck` class and related methods from the `deck.py` file - Add the `PythonDependencyRenderer` class in `renderer.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: use `TableRenderer` for rendering DataFrames - Import `TableRenderer` from a different module - Replace the `MarkdownRenderer` with `TableRenderer` to render a DataFrame as a table Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: refactor codebase for improved performance - Remove unused import statements in `flytekit/deck/renderer.py` - Add a new test for `PythonDependencyRenderer` in `tests/flytekit/unit/deck/test_deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: update test_deck.py for python dependency deck testing - Update test_deck.py to include python dependency deck in various test cases - Adjust expected deck counts in test cases to reflect the changes - Add test cases for scenarios involving python dependency deck and input and output decks Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize import statements for pandas in project files - Remove `import pandas as pd` and replace it with `import pandas as pandas` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor import statements for `TableRenderer` usage - Import `TableRenderer` from a different location in `flytekit/deck/renderer.py` - Remove `TableRenderer` from the imports in `flytekit-deck-standard/flytekitplugins/deck/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: update PythonDependencyRenderer description - Update the description of PythonDependencyDeck in PythonDependencyRenderer Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor type hints and assertions across files - Change the type hint for `df` parameter to `pandas.DataFrame` - Add an assertion to check the type of `df` is `pandas.DataFrame` Signed-off-by: jason.lai <jasonlai1218@gmail.com> * feat: refactor rendering classes in deck and plugins - Add TableRenderer to TimeLineDeck class in deck.py - Remove TableRenderer from deck/renderer.py - Add TableRenderer to PythonDependencyRenderer in deck/renderer.py - Add TableRenderer to flytekit-deck-standard plugin - Remove TableRenderer from test_renderer.py Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: standardize markdown formatting across files - Update the markdown table format in PythonDependencyRenderer class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor markdown table rendering in PythonDependencyRenderer - Update the markdown table format in the PythonDependencyRenderer class to use HTML table tags - Modify the markdown_table variable concatenation for installed packages entries in the PythonDependencyRenderer class Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: improve code consistency in PythonDependencyRenderer - Add a line break after the button element in PythonDependencyRenderer Signed-off-by: jason.lai <jasonlai1218@gmail.com> * style: standardize formatting for better readability - Modify the table headers to have left-aligned text - Add extra line breaks after the button element Signed-off-by: jason.lai <jasonlai1218@gmail.com> * fix: update assertion to check for `Name` and `Version` consistency - Change the assertion to check for `Name` and `Version` instead of `name` and `version` in the result Signed-off-by: jason.lai <jasonlai1218@gmail.com> * test: extend test deadlines across various tests - Increase the test deadlines from 2 to 20 seconds in test_type_conversion_errors.py - Update test deadlines from 2 to 40 seconds in test_type_conversion_errors.py - Adjust test deadlines in test_eager_workflows.py for various tests Signed-off-by: jason.lai <jasonlai1218@gmail.com> * Revert "test: extend test deadlines across various tests" This reverts commit 1885662. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Only run generate decks in python_function_task if decks are enabled Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove breakpoint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix test_deck.py tests to account for the number of expected decks (now that we're no longer writing decks unnecessarily) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Increase deadline of eager tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix lint error Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * test: update test assertions in test_deck.py - Add a patch to the `test_python_dependency_renderer` function - Include assertions for `numpy` and `1.21.0` in the test result - Update the `test_deck.py` file in the `tests/flytekit/unit/deck` directory Signed-off-by: jason.lai <jasonlai1218@gmail.com> * refactor: refactor Python code organization - Import `PythonDependencyRenderer` and `SourceCodeRenderer` in `PythonFunctionTask` from separate files - Modify the way `requirements_txt` is generated in `PythonDependencyRenderer` - Refactor the creation of the `table` variable in `PythonDependencyRenderer` to improve readability Signed-off-by: jason.lai <jasonlai1218@gmail.com> * docs: refactor project structure and improve user experience - Change the title of the deck from `Python Dependencies` to `Dependencies` - Update the error message in case of a subprocess error in fetching installed packages - Add a heading `Python Dependencies` above the table output in the HTML Signed-off-by: jason.lai <jasonlai1218@gmail.com> * Separate out tests that require hypothesis Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: jason.lai <jasonlai1218@gmail.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Jan Fiedler <jan@union.ai>

jasonlai1218 added 11 commits March 12, 2024 01:50

feat: refactor source code rendering in SourceCodeDeck class

fccb28d

- Add a new class `SourceCodeDeck` to render the source code of a task - Implement the `html` property in the `SourceCodeDeck` class Signed-off-by: jason.lai <jasonlai1218@gmail.com>

refactor: refactor class properties and initialization

4ec42a7

- Add a property method for `pythondependency_deck` - Initialize `python_dependency_deck` if it is None - Remove a property method for `timeline_deck` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

- feat: consolidate Python dependencies in flytekit/__init__.py

6e70e23

- Add `PythonDependencyDeck` to `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

refactor: refactor flytekit/deck/deck.py for pandas compatibility

4dfeb34

- Add `pandas` import to `flytekit/deck/deck.py` - Change the input of `TableRenderer().to_html` to a `pd.DataFrame` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

refactor: refactor return types across multiple files

27ec1f2

- Update the return type annotation for `pythondependency_deck` in `context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

feat: consolidate Python dependency management in FlyteContext

5fd853d

- Add `PythonDependencyDeck` to the TYPE_CHECKING import - Add a method `add_deck` to `FlyteContext` class Signed-off-by: jason.lai <jasonlai1218@gmail.com>

refactor: refactor code to use DataFrame for package handling

bc21848

- Refactor the code to use a DataFrame for installed packages handling Signed-off-by: jason.lai <jasonlai1218@gmail.com>

chore: refactor code for improved naming conventions

9b30dc6

- Update the name of the PythonDependencyDeck instance from "PythonDependencyDeck" to "Python Dependency" Signed-off-by: jason.lai <jasonlai1218@gmail.com>

style: improve table alignment styling in CSS

5ebc980

- Add CSS style to center align the table content Signed-off-by: jason.lai <jasonlai1218@gmail.com>

jasonlai1218 requested review from wild-endeavor, kumare3, eapolinario, pingsutw and cosmicBboy as code owners March 13, 2024 17:32

dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Mar 13, 2024

jasonlai1218 added 4 commits March 14, 2024 01:34

refactor: refactor imports in flytekit package

6ce4575

- Remove import of `PythonDependencyDeck` from `flytekit/__init__.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

refactor: consolidate import statements in core/context_manager.py

7556126

- Remove the import of `PythonDependencyDeck` from `flytekit/core/context_manager.py` - Add an import of `Deck` to `flytekit/core/context_manager.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

style: improve code consistency and error checking

534fa0f

- Add a condition to check for deck existence before appending - Change single quotes to double quotes for consistency - Update package split delimiter to double quotes Signed-off-by: jason.lai <jasonlai1218@gmail.com>

chore: optimize imports in deck.py files

363f84f

- Remove unnecessary import of `pandas` in `flytekit/deck/deck.py` - Add import of `pandas` in `flytekit/deck/deck.py` Signed-off-by: jason.lai <jasonlai1218@gmail.com>

pingsutw reviewed Mar 13, 2024

View reviewed changes

flytekit/deck/deck.py Outdated Show resolved Hide resolved

thomasjpfan reviewed Mar 13, 2024

View reviewed changes

flytekit/deck/deck.py Outdated Show resolved Hide resolved

flytekit/deck/deck.py Outdated Show resolved Hide resolved

eapolinario reviewed Mar 13, 2024

View reviewed changes

flytekit/deck/deck.py Outdated Show resolved Hide resolved

jasonlai1218 added 3 commits March 14, 2024 03:12

test: improve test coverage for PythonDependencyDeck class

a775c7f

- Add a new test for the PythonDependencyDeck class in test_deck.py - Add assertions for specific strings in the HTML content in the test_python_dependency_deck() function Signed-off-by: jason.lai <jasonlai1218@gmail.com>

feat: enhance user space deck management

a4773e0

- Clear user space decks before adding a new deck - Ensure only one deck is added to user space params Signed-off-by: jason.lai <jasonlai1218@gmail.com>

thomasjpfan reviewed Mar 13, 2024

View reviewed changes

flytekit/deck/deck.py Outdated Show resolved Hide resolved

flytekit/deck/deck.py Outdated Show resolved Hide resolved

style: standardize formatting for better readability

0aad322

- Modify the table headers to have left-aligned text - Add extra line breaks after the button element Signed-off-by: jason.lai <jasonlai1218@gmail.com>

thomasjpfan reviewed Mar 29, 2024

View reviewed changes

tests/flytekit/unit/deck/test_deck.py Outdated Show resolved Hide resolved

jasonlai1218 and others added 6 commits March 29, 2024 23:24

fix: update assertion to check for Name and Version consistency

7f40ad8

- Change the assertion to check for `Name` and `Version` instead of `name` and `version` in the result Signed-off-by: jason.lai <jasonlai1218@gmail.com>

Revert "test: extend test deadlines across various tests"

3a262c4

This reverts commit 1885662. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Only run generate decks in python_function_task if decks are enabled

1534070

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Remove breakpoint

03cb07c

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Fix test_deck.py tests to account for the number of expected decks (n…

6153065

…ow that we're no longer writing decks unnecessarily) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

jasonlai1218 requested a review from samhita-alla as a code owner March 29, 2024 23:23

eapolinario and others added 4 commits March 29, 2024 16:41

Increase deadline of eager tests

3574aad

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Fix lint error

edb7c18

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Merge branch 'master' into add-py-deps-deck

57597ed

thomasjpfan reviewed Apr 1, 2024

View reviewed changes

flytekit/core/python_function_task.py Outdated Show resolved Hide resolved

flytekit/deck/renderer.py Outdated Show resolved Hide resolved

flytekit/deck/renderer.py Outdated Show resolved Hide resolved

jasonlai1218 added 2 commits April 2, 2024 23:47

Merge branch 'master' into add-py-deps-deck

0f5a455

thomasjpfan reviewed Apr 2, 2024

View reviewed changes

jasonlai1218 added 2 commits April 3, 2024 11:24

Merge branch 'master' into add-py-deps-deck

8c9af6b

thomasjpfan previously approved these changes Apr 3, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by maintainer label Apr 3, 2024

Separate out tests that require hypothesis

814a952

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

eapolinario dismissed thomasjpfan’s stale review via 814a952 April 3, 2024 18:16

eapolinario approved these changes Apr 3, 2024

View reviewed changes

eapolinario merged commit eac772a into flyteorg:master Apr 3, 2024
48 of 49 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python Dependency Deck #2264

Add Python Dependency Deck #2264

jasonlai1218 commented Mar 13, 2024 •

edited

Loading

codecov bot commented Mar 13, 2024 •

edited

Loading

thomasjpfan left a comment

thomasjpfan Apr 2, 2024

jasonlai1218 Apr 3, 2024

thomasjpfan Apr 2, 2024

eapolinario Apr 2, 2024

eapolinario Apr 3, 2024 •

edited

Loading

thomasjpfan Apr 2, 2024 •

edited

Loading

thomasjpfan Apr 2, 2024

thomasjpfan Apr 2, 2024

jasonlai1218 Apr 3, 2024

thomasjpfan left a comment

eapolinario commented Apr 3, 2024 •

edited

Loading

	return ""
	return "Error occurred while fetching installed packages."

	python_dependencies_deck = Deck("Python Dependencies")
	python_dependencies_deck = Deck("Dependencies")

	def __init__(self, title: str = "Python Dependencies"):
	def __init__(self, title: str = "Dependencies"):

Add Python Dependency Deck #2264

Add Python Dependency Deck #2264

Conversation

jasonlai1218 commented Mar 13, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

codecov bot commented Mar 13, 2024 • edited Loading

Codecov Report

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan Apr 2, 2024

Choose a reason for hiding this comment

jasonlai1218 Apr 3, 2024

Choose a reason for hiding this comment

thomasjpfan Apr 2, 2024

Choose a reason for hiding this comment

eapolinario Apr 2, 2024

Choose a reason for hiding this comment

eapolinario Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

thomasjpfan Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

thomasjpfan Apr 2, 2024

Choose a reason for hiding this comment

thomasjpfan Apr 2, 2024

Choose a reason for hiding this comment

jasonlai1218 Apr 3, 2024

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

eapolinario commented Apr 3, 2024 • edited Loading

jasonlai1218 commented Mar 13, 2024 •

edited

Loading

codecov bot commented Mar 13, 2024 •

edited

Loading

eapolinario Apr 3, 2024 •

edited

Loading

thomasjpfan Apr 2, 2024 •

edited

Loading

eapolinario commented Apr 3, 2024 •

edited

Loading