Bug Or Feature - Experiment enters failed state but trials left running

[jlewi]

/kind bug

What steps did you take and what happened:
I'm not sure if this is a bug or a feature.

I had an experiment enter a failed state because too many trials failed.

All of the existing trials that were in the running state continued to run.

I can see advantages and disadvantages to not terminating existing trials when an experiment fails

[johnugeorge]

Currently, the behavior is same as you described. Do you have any particular use case?

[jlewi]

I don't have a specific use case to warrant one behavior or the other.

Training jobs can run for a long time so if training jobs have been running for 10 hours you don't necessarily want to just abort them; you might want to let them continue to run and finish.

I guess if the user wants to kill them they can always do explicitly. Wheras if they are killed automatically resuming the jobs may not be possible.

[gaocegege]

@johnugeorge Did we fix it in #861

[johnugeorge]

Yes. Currently, we keep existing running trials to continue and complete even if it has entered a terminal state.

[johnugeorge]

Closing the issue as it works as expected