Add Azure-specific headers when uploading to blob storage #1784
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
devictr
requested review from
wild-endeavor,
kumare3,
eapolinario,
pingsutw and
cosmicBboy
as code owners
|
Thank you for opening this pull request! 🙌 These tips will help get your PR across the finish line:
|
Signed-off-by: Victor Delépine <victor.delepine@wayve.ai>
eapolinario
reviewed
Aug 10, 2023
| verify=False | ||
| if self._config.platform.insecure_skip_verify is True | ||
| else self._config.platform.ca_cert_file_path, | ||
| ) | ||
|
|
||
| if rsp.status_code != requests.codes["OK"]: | ||
| if rsp.status_code not in (requests.codes["OK"], requests.codes["created"]): |
There was a problem hiding this comment.
why do we have to check this value as well?
There was a problem hiding this comment.
Ah sorry, forgot to specify in my PR description.
Azure returns an HTTP 201. So this was throwing an exception even though the request was successful
There was a problem hiding this comment.
can you add this as a comment please?
|
Maybe I should add a test on the entire |
pingsutw
reviewed
Aug 11, 2023
| @staticmethod | ||
| def get_extra_headers_for_protocol(native_url): | ||
| if native_url.startswith("abfs://"): | ||
| return {"x-ms-blob-type": "BlockBlob"} |
There was a problem hiding this comment.
In the case of blockblob type, the return code will become 200, won't it?
https://stackoverflow.com/questions/67015459/why-does-azure-blob-storage-return-201-but-not-upload
There was a problem hiding this comment.
Seems like it returns a 201 https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob?tabs=azure-ad#status-code
There was a problem hiding this comment.
That matches what happened when I tested it
Signed-off-by: Victor Delépine <victor.delepine@wayve.ai>
wild-endeavor
approved these changes
Aug 17, 2023
pingsutw
approved these changes
Aug 17, 2023
|
Congrats on merging your first pull request! 🎉 |
jeevb
pushed a commit
that referenced
this pull request
Sep 20, 2023
* Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> --------- Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Future-Outlier
pushed a commit
to Future-Outlier/flytekit
that referenced
this pull request
Oct 3, 2023
) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> --------- Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Future Outlier <eric901201@gmai.com>
jeevb
added a commit
that referenced
this pull request
Nov 1, 2023
* pip through to container Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move around Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add asserts Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * delete bad line Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add Azure-specific headers when uploading to blob storage (#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> --------- Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add async delete function in base_agent (#1800) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add support for execution name prefixes (#1803) Signed-off-by: troychiu <y.troychiu@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Remove ref in output (#1794) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Async file sensor (#1790) --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Eager workflows to support async workflows (#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up docs Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * more clean up Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * docs cleanup Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Enable SecretsManager.get to load and return bytes (#1798) * fix secretsmanager Signed-off-by: Yue Shang <s.yue3074@gmail.com> * fix lint issue Signed-off-by: Yue Shang <s.yue3074@gmail.com> * add doc Signed-off-by: Yue Shang <s.yue3074@gmail.com> * fix github check Signed-off-by: Yue Shang <s.yue3074@gmail.com> --------- Signed-off-by: Yue Shang <s.yue3074@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Batch upload flyte directory (#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <pingsutw@apache.org> * Update get method Signed-off-by: Kevin Su <pingsutw@apache.org> * Move batch size to type engine Signed-off-by: Kevin Su <pingsutw@apache.org> * comment Signed-off-by: Kevin Su <pingsutw@apache.org> * update comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Add test Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Better error messaging for overrides (#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Run remote Launchplan from `pyflyte run` (#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * one more improvement Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated command Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated formatting Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * bug fixed in types Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add is none function (#1757) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Dynamic workflow should not throw nested task warning (#1812) Signed-off-by: oliverhu <khu@linkedin.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add a manual image building GH action (#1816) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (#1813) Signed-off-by: Jan Fiedler <jan.fiedler@kineo.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * None doesnt work Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * unpartitioned selector Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix list of annotated structured dataset (#1817) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> * refactor variable name. Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> --------- Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Async agent delete function for while loop case (#1802) Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix docs warnings (#1827) Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix extract_task_module (#1829) --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Feat: Add type support for pydantic BaseModels (#1660) Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> Signed-off-by: Arthur <atte.book@gmail.com> Signed-off-by: wirthual <wirthra@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add test for unspecified mig Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add support for overriding accelerator Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move from core to extras Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Make FlyteRemote slightly more copy/pastable (#1830) Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pyflyte meta inputs (#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed unit test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Added Launch options Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixing docs failure Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Use mashumaro to serialize/deserialize dataclass (#1735) Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Co-authored-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Databricks Agent (#1797) Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Prometheus metrics (#1815) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pyflyte register optionally activates schedule (#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * comment addressed Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Remove versions 3.9 and 3.10 (#1831) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Snowflake agent (#1799) Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Update agent metric name (#1835) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * MemVerge MMCloud Agent (#1821) Signed-off-by: Edwin Yu <edwinyyyu@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add download badges in readme (#1836) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Eager local entrypoint and support for offloaded types (#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * wip local entrypoint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add local entrypoint tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * remove push step Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * update requirements and add snowflake agent to api reference (#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * update requirements Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove versions Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove tensorflow-macos Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * lint Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <aallasamhita@gmail.com> --------- Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix: Make sure decks created in elastic task workers are transferred to parent process (#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add accept grpc (#1841) * add accept grpc Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * default headers interceptor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * setup.py Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fmt Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move prometheus-client import Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Co-authored-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Address todo in docstring Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Create blank session if none provided Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add test for get_session Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Remove comments Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Minor corrections README.md Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> --------- Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Fix token client tests Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Fix plugin requirements Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> --------- Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * bump flyteidl Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make requirements Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix failing tests Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanups Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * update flyteidl Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * less verbose definitions Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Attempt at less confusing syntax Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Streamline UX Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Run make fmt Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: troychiu <y.troychiu@gmail.com> Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Yue Shang <s.yue3074@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: oliverhu <khu@linkedin.com> Signed-off-by: Jan Fiedler <jan.fiedler@kineo.ai> Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> Signed-off-by: Arthur <atte.book@gmail.com> Signed-off-by: wirthual <wirthra@gmail.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Victor Delépine <vctr.delepine@gmail.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Yi Chiu <114708546+troychiu@users.noreply.github.com> Co-authored-by: Matthew Hoffman <matthew@protopia.ai> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Yue Shang <138256885+ysysys3074@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Keqiu Hu <khu@linkedin.com> Co-authored-by: Jan Fiedler <89976021+fiedlerNr9@users.noreply.github.com> Co-authored-by: Chao-Heng Lee <chaohengstudent@gmail.com> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Arthur Böök <49250723+ArthurBook@users.noreply.github.com> Co-authored-by: Katrina Rogan <katroganGH@gmail.com> Co-authored-by: Po Han(Hank) Huang <hhcs9527@gmail.com> Co-authored-by: Edwin Yu <92917168+edwinyyyu@users.noreply.github.com> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
ringohoffman
added a commit
to ringohoffman/flytekit
that referenced
this pull request
Nov 24, 2023
* pip through to container Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move around Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add asserts Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * delete bad line Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add Azure-specific headers when uploading to blob storage (flyteorg#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> --------- Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add async delete function in base_agent (flyteorg#1800) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add support for execution name prefixes (flyteorg#1803) Signed-off-by: troychiu <y.troychiu@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Remove ref in output (flyteorg#1794) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (flyteorg#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Async file sensor (flyteorg#1790) --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Eager workflows to support async workflows (flyteorg#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up docs Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * more clean up Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * docs cleanup Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Enable SecretsManager.get to load and return bytes (flyteorg#1798) * fix secretsmanager Signed-off-by: Yue Shang <s.yue3074@gmail.com> * fix lint issue Signed-off-by: Yue Shang <s.yue3074@gmail.com> * add doc Signed-off-by: Yue Shang <s.yue3074@gmail.com> * fix github check Signed-off-by: Yue Shang <s.yue3074@gmail.com> --------- Signed-off-by: Yue Shang <s.yue3074@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Batch upload flyte directory (flyteorg#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <pingsutw@apache.org> * Update get method Signed-off-by: Kevin Su <pingsutw@apache.org> * Move batch size to type engine Signed-off-by: Kevin Su <pingsutw@apache.org> * comment Signed-off-by: Kevin Su <pingsutw@apache.org> * update comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> * Add test Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Better error messaging for overrides (flyteorg#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Run remote Launchplan from `pyflyte run` (flyteorg#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * one more improvement Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated command Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated formatting Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * bug fixed in types Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add is none function (flyteorg#1757) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Dynamic workflow should not throw nested task warning (flyteorg#1812) Signed-off-by: oliverhu <khu@linkedin.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add a manual image building GH action (flyteorg#1816) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (flyteorg#1813) Signed-off-by: Jan Fiedler <jan.fiedler@kineo.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * None doesnt work Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * unpartitioned selector Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix list of annotated structured dataset (flyteorg#1817) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (flyteorg#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> * refactor variable name. Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> --------- Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Async agent delete function for while loop case (flyteorg#1802) Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix docs warnings (flyteorg#1827) Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix extract_task_module (flyteorg#1829) --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Feat: Add type support for pydantic BaseModels (flyteorg#1660) Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> Signed-off-by: Arthur <atte.book@gmail.com> Signed-off-by: wirthual <wirthra@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add test for unspecified mig Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add support for overriding accelerator Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move from core to extras Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Make FlyteRemote slightly more copy/pastable (flyteorg#1830) Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pyflyte meta inputs (flyteorg#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed unit test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Added Launch options Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixing docs failure Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Use mashumaro to serialize/deserialize dataclass (flyteorg#1735) Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Co-authored-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Databricks Agent (flyteorg#1797) Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Prometheus metrics (flyteorg#1815) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pyflyte register optionally activates schedule (flyteorg#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * comment addressed Signed-off-by: Ketan Umare <ketan.umare@gmail.com> --------- Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Remove versions 3.9 and 3.10 (flyteorg#1831) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Snowflake agent (flyteorg#1799) Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Update agent metric name (flyteorg#1835) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * MemVerge MMCloud Agent (flyteorg#1821) Signed-off-by: Edwin Yu <edwinyyyu@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add download badges in readme (flyteorg#1836) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Eager local entrypoint and support for offloaded types (flyteorg#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * wip local entrypoint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add local entrypoint tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * clean up tests Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update ci Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * remove push step Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * update requirements and add snowflake agent to api reference (flyteorg#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * update requirements Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove versions Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove tensorflow-macos Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * lint Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <aallasamhita@gmail.com> --------- Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Fix: Make sure decks created in elastic task workers are transferred to parent process (flyteorg#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add accept grpc (flyteorg#1841) * add accept grpc Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * default headers interceptor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * setup.py Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fixes Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fmt Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move prometheus-client import Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Co-authored-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (flyteorg#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Address todo in docstring Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Create blank session if none provided Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add test for get_session Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Remove comments Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Minor corrections README.md Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> --------- Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Fix token client tests Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Fix plugin requirements Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> --------- Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * bump flyteidl Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make requirements Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix failing tests Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanup Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * cleanups Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * update flyteidl Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * less verbose definitions Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Attempt at less confusing syntax Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Streamline UX Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Run make fmt Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Victor Delépine <victor.delepine@wayve.ai> Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: troychiu <y.troychiu@gmail.com> Signed-off-by: Matthew Hoffman <matthew@protopia.ai> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Yue Shang <s.yue3074@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: oliverhu <khu@linkedin.com> Signed-off-by: Jan Fiedler <jan.fiedler@kineo.ai> Signed-off-by: Chao-Heng Lee <chaohengstudent@gmail.com> Signed-off-by: Adrian Rumpold <a.rumpold@gmail.com> Signed-off-by: Arthur <atte.book@gmail.com> Signed-off-by: wirthual <wirthra@gmail.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: HH <hhcs9527@gmail.com> Signed-off-by: hhcs9527 <hhcs9527@gmail.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Victor Delépine <vctr.delepine@gmail.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Yi Chiu <114708546+troychiu@users.noreply.github.com> Co-authored-by: Matthew Hoffman <matthew@protopia.ai> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Yue Shang <138256885+ysysys3074@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Keqiu Hu <khu@linkedin.com> Co-authored-by: Jan Fiedler <89976021+fiedlerNr9@users.noreply.github.com> Co-authored-by: Chao-Heng Lee <chaohengstudent@gmail.com> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Arthur Böök <49250723+ArthurBook@users.noreply.github.com> Co-authored-by: Katrina Rogan <katroganGH@gmail.com> Co-authored-by: Po Han(Hank) Huang <hhcs9527@gmail.com> Co-authored-by: Edwin Yu <92917168+edwinyyyu@users.noreply.github.com> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
This PR fixes an issue causing
pyflyte run --remoteto fail when using Azure.Type
Are all requirements met?
Complete description
The PUT call to Azure Blob Storage returned an HTTP 400 Error:
This is a starting point that unblocks me, but there's a discussion to be had regarding where you would like to handle such cases. Some open questions:
I have tested this locally by tweaking the class and trying to run a workflow
Tracking Issue
flyteorg/flyte#3942
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/