Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release v1.2] Backport v1.6 #1645

Merged
merged 81 commits into from
May 17, 2023
Merged

Conversation

eapolinario
Copy link
Collaborator

TL;DR

Please replace this text with a description of what this PR accomplishes.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

wild-endeavor and others added 30 commits May 16, 2023 14:00
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Remove retry from flytekit's setup.py and regenerate requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add to sagemaker

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove retry from sagemaker plugin requirements file

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Restore doc-requirements.txt

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Fix bad merge

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* fix: Make FlyteFile compatible with Annotated[..., HashMethod]

See issue #3424

Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>

* tests: Add test case for FlyteFile with HashMethod annotation

Issue: #3424
Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>

* fix: Use typing_extensions.Annotated for py3.8 compatibility

Issue: #3424
Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>

* fix: Use `get_args` and `get_origin` from typing_extensions for py3.8 compatibility

Issue: #3424
Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>

* fix(tests): Use fixture for local dummy file

Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>

---------

Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com>
* move FlyteSchema deprecation warning to initialization method

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* update

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

---------

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
…1515)

* add pod_template and pod_template_name arguments for ContainerTask

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

* factor out _serialize_pod_spec into separate util function

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

* model file changes, couple other changes

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* minor cleanup

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

* add unit test for container_task pod_template

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

* bump min version of flyteidl to 1.3.12

for pod template data config support

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

* require flyteidl==1.3.12 in doc-requirements.txt

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>

---------

Signed-off-by: Felix Ruess <felix.ruess@roboception.de>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: franco-bocci <franco.bocci@wolt.com>
* TypeTransformer for TensorFlow model

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* clean up

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* clean up

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* fix lint

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

---------

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Device auth flow

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Device AuthFlow is now available in flytekit

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* unit tests

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* test added

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Fixed test

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Fixed unit test

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Fixed lint errors

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

---------

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
* url encode secret first

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* nit

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
#1556)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Co-authored-by: ggydush-fn <greg.gydush@freenome.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Automatically remove unused import

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
* Disallow partial lists in map tasks

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Lint

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Convert List[Any] to a single pickle file

* remove redundant code

* keep batchSize only if type contain flytePickle

* fix error

* add batch support to translate_inputs_to_literals

* add ci test

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve comment

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* add more ci test

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* handle HashMethod case

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve format

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* improve

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* add test_is_batchable

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>

* Add BatchSize

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test_batch_size

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: root <root@ip-172-31-10-58.ec2.internal>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
* Unify sqlalchemy Dockerfiles

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Simplifyte sqlalchemy Dockerfile

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Set pythonpath to /root

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* pyflyte build

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Support serialize and package

Signed-off-by: Kevin Su <pingsutw@apache.org>

* more tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* move to plugin

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* fixed tested

Signed-off-by: Kevin Su <pingsutw@apache.org>

* fixed tested

Signed-off-by: Kevin Su <pingsutw@apache.org>

* more tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add support passing yaml in pyflyte run

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
…ror messages (#1582)

* improve input type conversion error

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix lint

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* add tests

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* add tests

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* add rich

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix lint

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* remove prototyping script, update loggers

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* update __init__.py

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* update logger

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* update logger

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix GE and pandera tests

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* fix lint

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

---------

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
* Add support nested FlyteFile in pyflyte run

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
* Set cache_regions=True in the case of s3fs

* Linting

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* Fix tests

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

* More linting

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
* set default ComSpec when running on Windows

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
* add activate-launchplan command to pyflyte

Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com>

* fix linting

Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com>

* rework launchplan command

Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com>

* fix linting

Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com>

---------

Signed-off-by: Artem Petrov <58334441+wckdman@users.noreply.github.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* checkpoint

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Experimental implementation works well. Instead of messing with the class, we utilize the interpolation and context to dynamically generate a wrapper around the desired script. In the wrapper, we cd to the ctx.working_directory, export the env variables (handled by an additional method to convert dict to str), and then pass the arguments to the script

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fix spacing

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* remove breakpoint

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Output ctx.working_directory as single output

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* more doc strings

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* more comments

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Added comments

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Add tests and test files from other branch

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* minor fix, have function return the instance rather than create. It seems to get registered when flyte packages your project

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fix tests

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* remove set flags not supported by sh

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fix spellcheck lint errors

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* don't have a windows equivalent test script so bypassing those tests for now

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Add typing to make_export_string_from_env_dict params

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fixup doc string

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Refactored the new behavior out into a new class. Did not change implementation details at all

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Address linter issues and up test cov

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Skip test on windows, no equivalent script

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Run black and isort, address SC2236

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Refactored class name to remove _, make utility function require name so multiple uses in a workflow don't break

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fix utility function

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* fix utility function call in tests

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* Fix isort on test_shell.py

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

* adding logging settings to papermill plugin

Signed-off-by: Calvin Leather <cleather@embarkvet.com>

* set level to info logging

* improved comments/docs

---------

Signed-off-by: Mike Zhong <mzhong@embarkvet.com>
Signed-off-by: Calvin Leather <cleather@embarkvet.com>
Co-authored-by: Mike Zhong <mzhong@embarkvet.com>
Co-authored-by: Mike Zhong <mike.zhong89@gmail.com>
pingsutw and others added 5 commits May 16, 2023 15:54
* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix mypy errors

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix mypy errors

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* wip

Signed-off-by: Kevin Su <pingsutw@apache.org>

* fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* fix test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update type

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update dev-requirements.txt

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Address comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* upgrade torch

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
@codecov
Copy link

codecov bot commented May 17, 2023

Codecov Report

❗ No coverage uploaded for pull request base (release-v1.2@f7b59bc). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             release-v1.2    #1645   +/-   ##
===============================================
  Coverage                ?   71.13%           
===============================================
  Files                   ?      331           
  Lines                   ?    30348           
  Branches                ?     2907           
===============================================
  Hits                    ?    21588           
  Misses                  ?     8208           
  Partials                ?      552           

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
This reverts commit af49155.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
@eapolinario eapolinario merged commit 4fd5d81 into release-v1.2 May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.