[FlyteCTL Feature] Schedule ergonomics

[wild-endeavor]

Improve scheduled workflow handling

Poor UX currently

There's a few ux issues around how scheduled workflows are handled today.

• When you register a new launch plan with a schedule, it doesn't become active without manual intervention. And there's no easy way to set this (like a --set-active or some option).
• When a user deletes a launch plan from code, and the user doesn't manually deactivate the launch plan, it will continue to run. There's also no way to query "old" launch plans.

As background info: For a given project, domain, and launch plan name, only one version can be "active" at any given time. Activating one version will deactivate the others.

New flytectl commands

Both of these can be addressed through flytectl. Proposal is to add some new commands

(not attached to the names of the commands here, just as examples)

activate-latest [--remove-old]
For all workflows, find the latest version. For each workflow at said version, activate all launchplans. This will automatically deactivate all other versions. If the remove-old switch is specified, then launch plans pointing to each workflow, but do not have a version matching the workflow's latest version, will also be deactivated.

activate --version <v>
Activate all launch plans for a project/domain at a given version if it exists.

deactivate-all
Deactivate all launch plans across a project domain. I think this is helpful for ensuring that nothing is

Docs

We also need to improve documentation around how handling of schedules should work in prod.

Misc

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[EngHabu]

I'm all for these commands:
Activate all launch plans at a specific version:

flytectl update launchplan --activate --version <v> --all

Latest:

flytectl update launchplan --activate --latest --all

Deactivate all launch plans [at a specific version]:

flytectl update launchplan --archive [--version <v>] --all

What I'm struggling with is the --remove-old part... There is an inherit assumption here that all things registered to a given project belong to the same "source tree" to a degree. Right?
We really want to detect things that used to get registered together but no longer are (i.e. deleted), I think the maybe right semantics here are:

• When pyflyte register / flytectl register, we register the archive not individual files (which I think is the case)
• We record the source tree/archive file path/checksum... etc. along with these entities as a label/tag
• We can then find all items that belong to the same source tree when --remove-old is passed..
  Do you agree?

Some open questions:

• What happens to the ad-hoc created launchplans and the future-UI created LPs?
• Or do we really mean only LPs registered to workflows referenced by the new/updated LPs? Then commands above will need to change...

[pmahindrakar-oss]

BTW Flytectl register file has an option to enable schedules which are getting registered and this will activate the version which is being registered and deactivate the rest versions belonging to the same launchplan

Enable schedule for the launchplans part of the serialized protobuf files:

 flytectl register file  _pb_output/* -d development  -p flytesnacks --version v2 --enableSchedule


      --enableSchedule                Enable the schedule if the files contain schedulable launchplan.

[davidmirror-ops]

Related
As indicated by @RyanKor on this thread:
"the function I expect is the activated launchplans should be archived when the projects they are included become archived"

[kumare3]

I am adding a new pyflyte register --flip-activate command that will toggle all old launch plans to deactivated and activate all the new ones

[kumare3]

many other features can be added to pyflyte launchplan, much faster to iterate