Skip to content
This repository has been archived by the owner on Oct 23, 2023. It is now read-only.

Backport 1.7 to 1.2 #420

Merged
merged 24 commits into from
Jun 28, 2023
Merged

Backport 1.7 to 1.2 #420

merged 24 commits into from
Jun 28, 2023

Commits on Jun 27, 2023

  1. Adding support for structured dataset (#369) 

    Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>
    @pmahindrakar-oss @eapolinario
    pmahindrakar-oss authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    b7c6bec View commit details
    Browse the repository at this point in the history

  2. added dynamic_job_spec_uri to dynamic workflow metadata and node exec… 

    …ution closure (#360)

Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw @eapolinario
    hamersaw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    71836cc View commit details
    Browse the repository at this point in the history

  3. Use TokenCache in ClientCredentialsTokenSourceProvider (#377)

    @andrewwdye @eapolinario
    andrewwdye authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    5af6464 View commit details
    Browse the repository at this point in the history

  4. Init customTokenSource.refreshTime (#381) 

    Signed-off-by: Andrew Dye <andrewwdye@gmail.com>
    @andrewwdye @eapolinario
    andrewwdye authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    d059a27 View commit details
    Browse the repository at this point in the history

  5. added DataLoadingConfig to K8sPod message (#368) 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw @eapolinario
    hamersaw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    73e49e3 View commit details
    Browse the repository at this point in the history

  6. Add Reasons field to TaskExecutionClosure to track time-series of rea… 

    …sons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added docs

Signed-off-by: Daniel Rammer <daniel@union.ai>

* actually finishing docs too

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw @eapolinario
    hamersaw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    018aa0d View commit details
    Browse the repository at this point in the history

  7. Create service for runtime metrics (#367) 

    * added span messages

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added endpoints to service

Signed-off-by: Daniel Rammer <daniel@union.ai>

* generated mocks

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed get task execution metrics rpc

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added EXECUTION_IDLE category

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated PLUGIN_EXECUTION to PLUGIN_RUNTIME

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed recorded_at on workflow and node level events

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added docs for task event reported_at field

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed GetNodeExecutionMetrics endpoint - will implement later if necessary

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated docs

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added reported_at for node execution events

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed typo

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed typos and removed dead code

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated categories

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added workflow setup and teardown categories

Signed-off-by: Daniel Rammer <daniel@union.ai>

* simplified span message and moved to flyteidl.core

Signed-off-by: Daniel Rammer <daniel@union.ai>

---------

Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw @eapolinario
    hamersaw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    3edc059 View commit details
    Browse the repository at this point in the history

  8. Remove misleading token refresh logic from client credentials token s… 

    …ource provider (#383)
    @andrewwdye @eapolinario
    andrewwdye authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    234627c View commit details
    Browse the repository at this point in the history

  9. Out of core plugin (#378) 

    * Add backend plugin system service

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add backend plugin system service

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update state

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update state

Signed-off-by: Kevin Su <pingsutw@apache.org>

* dics

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove output prefix from get request

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update

Signed-off-by: Kevin Su <pingsutw@apache.org>

* remove prev state

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update proto

Signed-off-by: Kevin Su <pingsutw@apache.org>

* remove error message

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* make generate

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Rename the service

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
    @pingsutw @eapolinario
    pingsutw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    f8e622e View commit details
    Browse the repository at this point in the history

  10. Feat: Add ElasticConfig message type for torch elastic training (#394) 

    * Add elastic config args to pytorch proto

Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com>

* Add elastic config message type for torchrun training

Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com>

---------

Signed-off-by: Fabio Graetz <fabiograetz@googlemail.com>
Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com>
Co-authored-by: Ketan Umare <ketan.umare@gmail.com>
    @fg91 @kumare3 @eapolinario
    3 people authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    44cbad6 View commit details
    Browse the repository at this point in the history

  11. Retract 1.4.x (#397) 

    Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
    @eapolinario
    eapolinario and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    a1fbb23 View commit details
    Browse the repository at this point in the history

  12. Data addresses #minor (#391) 

    Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
    @wild-endeavor @eapolinario
    wild-endeavor authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    5a2e346 View commit details
    Browse the repository at this point in the history

  13. Refactor kf-operator plugins configs and support setting different sp… 

    …ecs for different replica groups (#386)

* refactor kubeflow operators proto

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add back the original proto for backward compatible

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* clean up comments

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add kubeflow.rs

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add elastic config

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add command to MPI

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* add slots and command to mpi spec

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

---------

Signed-off-by: Yubo Wang <yubwang@linkedin.com>
Co-authored-by: Yubo Wang <yubwang@linkedin.com>
    @yubofredwang @eapolinario
    2 people authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e584dd9 View commit details
    Browse the repository at this point in the history

  14. add user_identifier (#388) 

    Signed-off-by: byhsu <byhsu@linkedin.com>
Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com>
    @ByronHsu @eapolinario
    3 people committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    2ceb09e View commit details
    Browse the repository at this point in the history

  15. Add envs to execution spec (#400) 

    Signed-off-by: Kevin Su <pingsutw@apache.org>
    @pingsutw @eapolinario
    pingsutw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    49bd1f0 View commit details
    Browse the repository at this point in the history

  16. Support union and none type in flyteidl (#401) 

    * add support for Union Scalar

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* support union type and literals

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

* change union type extraction

Signed-off-by: Yubo Wang <yubwang@linkedin.com>

---------

Signed-off-by: Yubo Wang <yubwang@linkedin.com>
Co-authored-by: Yubo Wang <yubwang@linkedin.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
    @yubofredwang @pingsutw @eapolinario
    3 people authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    ba1102d View commit details
    Browse the repository at this point in the history

  17. Rename user_identity to execution_identity (#402) 

    Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
    @ByronHsu @eapolinario
    2 people authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    7d2508f View commit details
    Browse the repository at this point in the history

  18. Single literal in GetDataResponse (#404) 

    Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
    @wild-endeavor @eapolinario
    wild-endeavor authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    ee1a8b7 View commit details
    Browse the repository at this point in the history

  19. Add namespace to execution system metadata (#406) 

    Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
    @katrogan @eapolinario
    katrogan authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    76ae152 View commit details
    Browse the repository at this point in the history

  20. Add oauth2 http proxy client (#405) 

    Signed-off-by: byhsu <byhsu@linkedin.com>
    @ByronHsu @eapolinario
    ByronHsu authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e9fe585 View commit details
    Browse the repository at this point in the history

  21. Rename externalPluginService to AgentService (#410) 

    * Rename externalPluginService to AgentService

Signed-off-by: Kevin Su <pingsutw@apache.org>

* nit

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
    @pingsutw @eapolinario
    pingsutw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    7d63ef4 View commit details
    Browse the repository at this point in the history

  22. Add external_plugin_service proto back to the idl (#413) 

    * Add external-plugin-service proto back to the idl

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update idl

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update idll

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update idll

Signed-off-by: Kevin Su <pingsutw@apache.org>

* AsyncAgentService

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
    @pingsutw @eapolinario
    pingsutw authored and eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e3ed27b View commit details
    Browse the repository at this point in the history

  23. Rerun make generate 

    Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
    @eapolinario
    eapolinario committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    0293e86 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Merge release-v1.2 and regenerate stubs 

    Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com>
    @eapolinario
    eapolinario committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    c6d8dc6 View commit details
    Browse the repository at this point in the history