Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Python Dependency Deck #2264

Merged
merged 66 commits into from
Apr 3, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
fccb28d
feat: refactor source code rendering in `SourceCodeDeck` class
jasonlai1218 Mar 11, 2024
1fec103
refactor: refactor deck rendering for Python dependencies
jasonlai1218 Mar 12, 2024
4ec42a7
refactor: refactor class properties and initialization
jasonlai1218 Mar 12, 2024
6e70e23
- feat: consolidate Python dependencies in `flytekit/__init__.py`
jasonlai1218 Mar 12, 2024
4dfeb34
refactor: refactor `flytekit/deck/deck.py` for `pandas` compatibility
jasonlai1218 Mar 12, 2024
27ec1f2
refactor: refactor return types across multiple files
jasonlai1218 Mar 12, 2024
5fd853d
feat: consolidate Python dependency management in FlyteContext
jasonlai1218 Mar 12, 2024
bc21848
refactor: refactor code to use DataFrame for package handling
jasonlai1218 Mar 13, 2024
9b30dc6
chore: refactor code for improved naming conventions
jasonlai1218 Mar 13, 2024
5ebc980
style: improve table alignment styling in CSS
jasonlai1218 Mar 13, 2024
501fcb6
refactor: refactor method and variable names across files
jasonlai1218 Mar 13, 2024
6ce4575
refactor: refactor imports in flytekit package
jasonlai1218 Mar 13, 2024
7556126
refactor: consolidate import statements in core/context_manager.py
jasonlai1218 Mar 13, 2024
534fa0f
style: improve code consistency and error checking
jasonlai1218 Mar 13, 2024
c4358f0
refactor: refactor Python dependency handling in classes
jasonlai1218 Mar 13, 2024
363f84f
chore: optimize imports in deck.py files
jasonlai1218 Mar 13, 2024
a775c7f
test: improve test coverage for PythonDependencyDeck class
jasonlai1218 Mar 13, 2024
a4773e0
feat: enhance user space deck management
jasonlai1218 Mar 13, 2024
05db9ab
refactor: refactor deck module and unit tests
jasonlai1218 Mar 13, 2024
d2bc23b
fix: update subprocess calls to use `sys.executable`
jasonlai1218 Mar 16, 2024
099a931
feat: refactor HTML generation logic and improve user experience
jasonlai1218 Mar 16, 2024
b02db7c
feat: enhance table content copying functionality
jasonlai1218 Mar 16, 2024
aaedf7a
refactor: improve table content copying functionality
jasonlai1218 Mar 16, 2024
938e0dd
refactor: improve package management and error handling
jasonlai1218 Mar 16, 2024
265cc60
chore: standardize whitespace in requirements_txt handling
jasonlai1218 Mar 16, 2024
7e132c0
style: standardize quotation marks for package_info keys
jasonlai1218 Mar 16, 2024
9eacfe7
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 16, 2024
20a3e9f
refactor: simplify requirements_txt generation
jasonlai1218 Mar 17, 2024
341a7da
docs: fix typos and improve code consistency across files
jasonlai1218 Mar 17, 2024
de90326
refactor: refactor dependency handling in PythonDependencyDeck class
jasonlai1218 Mar 18, 2024
b5f49b4
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 18, 2024
5634e78
refactor: update default name and test assertion in PythonDependencyDeck
jasonlai1218 Mar 18, 2024
9bd7995
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 19, 2024
79a09ac
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 20, 2024
e4c0bd0
refactor: update rendering of Pandas DataFrame using MarkdownRenderer
jasonlai1218 Mar 25, 2024
0091f25
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 25, 2024
5a58a8e
refactor: refactor PythonDependencyDeck and related classes
jasonlai1218 Mar 25, 2024
12ae643
feat: use `TableRenderer` for rendering DataFrames
jasonlai1218 Mar 26, 2024
16e681c
test: refactor codebase for improved performance
jasonlai1218 Mar 26, 2024
fc21591
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 26, 2024
ac21522
test: update test_deck.py for python dependency deck testing
jasonlai1218 Mar 26, 2024
96a852c
style: standardize import statements for pandas in project files
jasonlai1218 Mar 26, 2024
20b84fc
refactor: refactor import statements for `TableRenderer` usage
jasonlai1218 Mar 27, 2024
8572008
refactor: update PythonDependencyRenderer description
jasonlai1218 Mar 27, 2024
1c93057
refactor: refactor type hints and assertions across files
jasonlai1218 Mar 27, 2024
a32ce2d
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 27, 2024
8a40607
feat: refactor rendering classes in deck and plugins
jasonlai1218 Mar 29, 2024
f731757
docs: standardize markdown formatting across files
jasonlai1218 Mar 29, 2024
20e96c5
refactor: refactor markdown table rendering in PythonDependencyRenderer
jasonlai1218 Mar 29, 2024
352b79b
style: improve code consistency in PythonDependencyRenderer
jasonlai1218 Mar 29, 2024
0aad322
style: standardize formatting for better readability
jasonlai1218 Mar 29, 2024
7f40ad8
fix: update assertion to check for `Name` and `Version` consistency
jasonlai1218 Mar 29, 2024
1885662
test: extend test deadlines across various tests
jasonlai1218 Mar 29, 2024
3a262c4
Revert "test: extend test deadlines across various tests"
eapolinario Mar 29, 2024
1534070
Only run generate decks in python_function_task if decks are enabled
eapolinario Mar 29, 2024
03cb07c
Remove breakpoint
eapolinario Mar 29, 2024
6153065
Fix test_deck.py tests to account for the number of expected decks (n…
eapolinario Mar 29, 2024
3574aad
Increase deadline of eager tests
eapolinario Mar 29, 2024
edb7c18
Fix lint error
eapolinario Mar 30, 2024
57597ed
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Mar 30, 2024
9a2470e
test: update test assertions in test_deck.py
jasonlai1218 Mar 30, 2024
fc81271
refactor: refactor Python code organization
jasonlai1218 Apr 2, 2024
0f5a455
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Apr 2, 2024
ef47b14
docs: refactor project structure and improve user experience
jasonlai1218 Apr 3, 2024
8c9af6b
Merge branch 'master' into add-py-deps-deck
jasonlai1218 Apr 3, 2024
814a952
Separate out tests that require hypothesis
eapolinario Apr 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions flytekit/core/context_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,17 @@ def get_deck(self) -> typing.Union[str, "IPython.core.display.HTML"]: # type:ig

return _get_deck(typing.cast(ExecutionState, self.execution_state).user_space_params)

def add_deck(self, deck: Deck) -> None:
eapolinario marked this conversation as resolved.
Show resolved Hide resolved
"""
Adds a deck to the current context.
"""
if (
self.execution_state is not None
and self.execution_state.user_space_params is not None
and hasattr(self.execution_state.user_space_params, "decks")
):
self.execution_state.user_space_params.decks.append(deck)

@dataclass
class Builder(object):
file_access: FileAccessProvider
Expand Down
26 changes: 26 additions & 0 deletions flytekit/deck/deck.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,29 @@
autoescape=select_autoescape(enabled_extensions=("html",)),
)
return env.get_template("template.html")


class PythonDependencyDeck(Deck):
"""
PythonDependencyDeck is a deck that contains information about the library name and version.
"""

def __init__(self, name: Optional[str] = "Python Dependency", html: Optional[str] = ""):
eapolinario marked this conversation as resolved.
Show resolved Hide resolved
super().__init__(name, html)

@property
def html(self) -> str:
import json
import subprocess

Check warning on line 190 in flytekit/deck/deck.py

View check run for this annotation

Codecov / codecov/patch

flytekit/deck/deck.py#L189-L190

Added lines #L189 - L190 were not covered by tests

import pandas as pd
from flytekitplugins.deck.renderer import TableRenderer

Check warning on line 193 in flytekit/deck/deck.py

View check run for this annotation

Codecov / codecov/patch

flytekit/deck/deck.py#L192-L193

Added lines #L192 - L193 were not covered by tests
pingsutw marked this conversation as resolved.
Show resolved Hide resolved

try:
installed_packages = json.loads(subprocess.check_output(["pip", "list", "--format", "json"]))
eapolinario marked this conversation as resolved.
Show resolved Hide resolved
jasonlai1218 marked this conversation as resolved.
Show resolved Hide resolved
except subprocess.CalledProcessError as e:
logger.error(f"Error occurred while fetching installed packages: {e}")
return ""
df = pd.DataFrame(installed_packages)
html = TableRenderer().to_html(df)
jasonlai1218 marked this conversation as resolved.
Show resolved Hide resolved
return html

Check warning on line 202 in flytekit/deck/deck.py

View check run for this annotation

Codecov / codecov/patch

flytekit/deck/deck.py#L195-L202

Added lines #L195 - L202 were not covered by tests
11 changes: 10 additions & 1 deletion tests/flytekit/unit/deck/test_deck.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import flytekit
from flytekit import Deck, FlyteContextManager, task
from flytekit.deck import MarkdownRenderer, SourceCodeRenderer, TopFrameRenderer
from flytekit.deck.deck import _output_deck
from flytekit.deck.deck import PythonDependencyDeck, _output_deck


@pytest.mark.skipif("pandas" not in sys.modules, reason="Pandas is not installed.")
Expand Down Expand Up @@ -176,3 +176,12 @@ def test_source_code_renderer():
# Assert that the color #ffffff is used instead of #fff0f0
assert "#ffffff" in result
assert "#fff0f0" not in result


def test_python_dependency_deck():
python_dependency_deck = PythonDependencyDeck()
ctx = FlyteContextManager.current_context()
ctx.user_space_params.decks.clear()
ctx.add_deck(python_dependency_deck)
assert len(ctx.user_space_params.decks) == 1
assert ctx.user_space_params.decks[0].name == "Python Dependency"
Loading