Backport 1.7 to 1.2 #420

eapolinario · 2023-06-27T23:12:07Z

TL;DR

Backport 1.7 changes to 1.2 release branch

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

Second to last backport of releases to the 1.2 release branch, so that users of protobuf 3 can benefit from new features.

List of backported changes:

Tracking Issue

Remove the 'fixes' keyword if there will be multiple PRs to fix the linked issue

fixes https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

…ution closure (#360) Signed-off-by: Daniel Rammer <daniel@union.ai>

Signed-off-by: Andrew Dye <andrewwdye@gmail.com>

Signed-off-by: Daniel Rammer <daniel@union.ai>

…sons (#382) * added a time-series of reasons to the TaskExecution closure Signed-off-by: Daniel Rammer <daniel@union.ai> * added docs Signed-off-by: Daniel Rammer <daniel@union.ai> * actually finishing docs too Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai>

* added span messages Signed-off-by: Daniel Rammer <daniel@union.ai> * added endpoints to service Signed-off-by: Daniel Rammer <daniel@union.ai> * generated mocks Signed-off-by: Daniel Rammer <daniel@union.ai> * removed get task execution metrics rpc Signed-off-by: Daniel Rammer <daniel@union.ai> * added EXECUTION_IDLE category Signed-off-by: Daniel Rammer <daniel@union.ai> * updated PLUGIN_EXECUTION to PLUGIN_RUNTIME Signed-off-by: Daniel Rammer <daniel@union.ai> * removed recorded_at on workflow and node level events Signed-off-by: Daniel Rammer <daniel@union.ai> * added docs for task event reported_at field Signed-off-by: Daniel Rammer <daniel@union.ai> * removed GetNodeExecutionMetrics endpoint - will implement later if necessary Signed-off-by: Daniel Rammer <daniel@union.ai> * updated docs Signed-off-by: Daniel Rammer <daniel@union.ai> * added reported_at for node execution events Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typo Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typos and removed dead code Signed-off-by: Daniel Rammer <daniel@union.ai> * updated categories Signed-off-by: Daniel Rammer <daniel@union.ai> * added workflow setup and teardown categories Signed-off-by: Daniel Rammer <daniel@union.ai> * simplified span message and moved to flyteidl.core Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai>

…ource provider (#383)

* Add backend plugin system service Signed-off-by: Kevin Su <pingsutw@apache.org> * Add backend plugin system service Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * update state Signed-off-by: Kevin Su <pingsutw@apache.org> * update state Signed-off-by: Kevin Su <pingsutw@apache.org> * dics Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove output prefix from get request Signed-off-by: Kevin Su <pingsutw@apache.org> * update Signed-off-by: Kevin Su <pingsutw@apache.org> * remove prev state Signed-off-by: Kevin Su <pingsutw@apache.org> * update proto Signed-off-by: Kevin Su <pingsutw@apache.org> * remove error message Signed-off-by: Kevin Su <pingsutw@apache.org> * update comment Signed-off-by: Kevin Su <pingsutw@apache.org> * make generate Signed-off-by: Kevin Su <pingsutw@apache.org> * Rename the service Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> --------- Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Ketan Umare <ketan.umare@gmail.com>

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

…ecs for different replica groups (#386) * refactor kubeflow operators proto Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add back the original proto for backward compatible Signed-off-by: Yubo Wang <yubwang@linkedin.com> * clean up comments Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add kubeflow.rs Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add elastic config Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add command to MPI Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add slots and command to mpi spec Signed-off-by: Yubo Wang <yubwang@linkedin.com> --------- Signed-off-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com>

Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add support for Union Scalar Signed-off-by: Yubo Wang <yubwang@linkedin.com> * support union type and literals Signed-off-by: Yubo Wang <yubwang@linkedin.com> * change union type extraction Signed-off-by: Yubo Wang <yubwang@linkedin.com> --------- Signed-off-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com>

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

Signed-off-by: byhsu <byhsu@linkedin.com>

* Rename externalPluginService to AgentService Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add external-plugin-service proto back to the idl Signed-off-by: Kevin Su <pingsutw@apache.org> * update idl Signed-off-by: Kevin Su <pingsutw@apache.org> * update idll Signed-off-by: Kevin Su <pingsutw@apache.org> * update idll Signed-off-by: Kevin Su <pingsutw@apache.org> * AsyncAgentService Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

pmahindrakar-oss and others added 24 commits June 27, 2023 15:33

Adding support for structured dataset (#369)

b7c6bec

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

added dynamic_job_spec_uri to dynamic workflow metadata and node exec…

71836cc

…ution closure (#360) Signed-off-by: Daniel Rammer <daniel@union.ai>

Use TokenCache in ClientCredentialsTokenSourceProvider (#377)

5af6464

Init customTokenSource.refreshTime (#381)

d059a27

Signed-off-by: Andrew Dye <andrewwdye@gmail.com>

added DataLoadingConfig to K8sPod message (#368)

73e49e3

Signed-off-by: Daniel Rammer <daniel@union.ai>

Remove misleading token refresh logic from client credentials token s…

234627c

…ource provider (#383)

Retract 1.4.x (#397)

a1fbb23

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>

Data addresses #minor (#391)

5a2e346

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

add user_identifier (#388)

2ceb09e

Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>

Add envs to execution spec (#400)

49bd1f0

Signed-off-by: Kevin Su <pingsutw@apache.org>

Rename user_identity to execution_identity (#402)

7d2508f

Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com>

Single literal in GetDataResponse (#404)

ee1a8b7

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Add namespace to execution system metadata (#406)

76ae152

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

Add oauth2 http proxy client (#405)

e9fe585

Signed-off-by: byhsu <byhsu@linkedin.com>

Rename externalPluginService to AgentService (#410)

7d63ef4

* Rename externalPluginService to AgentService Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org>

Rerun make generate

0293e86

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

Merge release-v1.2 and regenerate stubs

c6d8dc6

Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>

wild-endeavor approved these changes Jun 28, 2023

View reviewed changes

eapolinario merged commit 635f2e6 into release-v1.2 Jun 28, 2023

eapolinario deleted the backport-1.7-to-1.2 branch June 28, 2023 19:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport 1.7 to 1.2 #420

Backport 1.7 to 1.2 #420

eapolinario commented Jun 27, 2023 •

edited

Loading

Backport 1.7 to 1.2 #420

Backport 1.7 to 1.2 #420

Conversation

eapolinario commented Jun 27, 2023 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

eapolinario commented Jun 27, 2023 •

edited

Loading