[Umbrella] Support Suspend in volcano

[kerthcet]

What is the problem you're trying to solve

We would like to make vcjob part of the Kueue ecosystem working as a high level job queueing component. Meanwhile, in preemption scenarios, suspending the job (or other terms in volcano) is somehow a foundational capacity.

Describe the solution you'd like

Make it possible to suspend the vcjob and reclaim the owned Pods. This can be achieved via two ways:

• By abort command, however, there're several questions we need to solve, list here:
  • suspend/resume 3 times vcjob will directly failure #3067
  • Pod conditions are not aggregated
• By introducing a new field just like batch job.spec.suspend

I would like to leverage the existing function rather than reinventing the wheel if possible.

Additional context

related issues:

• suspend/resume 3 times vcjob will directly failure #3067
• Support suspend like a batch job #2785

[kerthcet]

cc @GhangZh

[googs1025]

+1, this will be a very useful feature 👍

[Monokaix]

Hi, Thanks for your contribution! Just a little confused here, I'd like to know why kueue needs vcjob suspension. kueue is an independent queue management project, and Volcano also has its own queue, Volcano's own scheduling and queue are integrated. Why do we need to adapt a separate queue project?

[kerthcet]

Thanks @Monokaix for the inputs and sorry for the incomplete context, I'll explain more clearly here.

First of all, I think we can come to a consequence that Suspend(or other terminologies) is somehow a necessary capacity for Jobs(batch job, vcjob, rayjob, etc.) because of the limited resources that can't satisfy the job at the moment but will be met later with resources reclaimed. And I do think volcano supports this already, as I mentioned the cli suspend/resume capacity.

And the second question about why kueue needs this is just because asked by our users and community partners, they use both volcano and other schedulers in their clusters, they hope then can have a global job queueing system in the front. And they do have a forked volcano running in their clusters to finish this work.

Based on this, I do think this is reasonable to support Suspend better in volcano. And actually I do have several questions before we come to the proposal and I'm not sure about the answer, maybe someone can help me to understand this better:

• How preemption implements in volcano, I think generally it invokes the evict function and update the task status to Releasing?
• When resources not ready, the job will be marked as pending, right?
• Besides cli suspend, does the Aborted state has other scenarios? Like really failures happened here.

[kerthcet]

FYI: I tried to update the vcjob with status.state.phase = Aborted, it doesn't work. It will be rolled to Running.

[kerthcet]

Another problem is once resumed, the completed ones will be ignored which means we'll restart a fairly new task.

[hwdef]

I think jobTemplate alse support this scenario.
You hope that vcjob will not be processed after it is created. This is exactly the goal of jobTemplate.
jobTemplate has the same fields as vcjob,
You can create a controller and create a vcjob based on jobtemplate when receiving a jobtemplate creation request.

This way you don’t need to change any API of volcano, you just need to create an external controller.

[kerthcet]

Thanks @hwdef Any document about jobTemplate, what's this used for in volcano? Will vcjob controller watch for this resource?

[hwdef]

Please check this:
https://github.com/volcano-sh/volcano/tree/master/docs/design/jobflow#introduction-2

It is the template of vcjob and can be referenced by jobflow
If you don't create jobflow and associate jobtemplate, then no controller will handle jobtemplate

[kerthcet]

Can jobtemplate be rolled back? Suspend means a called job could be switched between running or pending with arbitrary times, I guess jobtemplate could not, because it's a flow.