Fair-Share scheduler improvements

[Tansito]

For future reads the context comes from this PR: #570

There are two parts where I would like to see improvements over the current implementation: structure, architecture.

Structure

At this moment, all the logic is fully integrated with the API application. I would like to propose a little refactor around the scheduler, moving it out from the API to a new application called scheduler. My main purpose with this change is to separate responsibilities and scopes. From my perspective this will make the code be easier to follow.

Architecture

The current implementation has a combination of django-commands plus polling loop that can be difficult to follow & maintain. To solve this problem we have different approaches. An implementation that I was thinking is based in event-driven using a combination of django-signals plus celery. This implementation would have some core keys:

• We will be able to send async tasks/events to the celery scheduler with the new job to be executed by Ray. Implementing a near approach to AoT scheduler.
• Instead to have the current loop we could use signals to notify when a resource is removed and there is capacity for more.

Something that I took in consideration when I was analyzing different solutions is that due to the Ray's decentralized scheduler can be tricky to analyze the status of Ray (I leave some references here):

• Ray queue details required  ray-project/ray#10644
• https://docs.ray.io/en/latest/ray-core/scheduling/memory-management.html#memory-aware-scheduling
• https://docs.ray.io/en/latest/ray-observability/state/state-api.html#garbage-collected-resources

That is one of the main reasons because I couldn't attach the task/event creation to the ray's job finish. I continue investigating this approach though:

• https://docs.ray.io/en/latest/ray-observability/state/state-api.html#monitoring-ray-states

I would like to hear opinions from you too: @pacomf , @IceKhan13 , @psschwei , @akihikokuroda 😄

[IceKhan13]

Fully support both statements:

• decouple scheduling and api
• migrate to event driven architecture

---

Pros and cons

Decouple scheduling and api:

• Pros:
  • separation of concerns
• Cons:
  • additional application requires additional management

Migrate to event driven architecture:

• Pros:
  • scalability
  • loose coupling (will tackle decouple scheduling and api)
• Cons:
  • fair share event scheduling / queueing is not there yet usually it is FIFO thing
  • ray does not support jobs callabacks, so we do not know when job is finished (so we need polling until this feature is supported)
    • I think this is the main thing here which stopped us to for this option in a first place (and time ofc :) )
  • complexity, it is known that event driven systems are harded to manage
  • debugging trickier too

---

We initially thought to go that route, but lended up with current implementation due to some reasons :) one of them is time to implementation. But eventually we need to migrate to that, for sure!

[pacomf]

I agree and the data event oriented is a must

[psschwei]

On structure: I agree in separating into separate apps

On architecture: I'm a bit lukewarm here. Do we have the volume to necessitate a move to an event-driven architecture? Assuming for sake of argument that we stick with a polling approach for now, how easy would it be to migrate to an event-driven approach in the future if we did need to make that move? All things being equal, I'd advocate for keeping the implementation as simple to support our workloads.

[Tansito]

Related with my answer to @IceKhan13 here, I agree with you @psschwei . I would say that it's an option to discuss middle improvements like cron-tasks instead of a while executed by script, for example.

[psschwei]

makes sense... I think I might go with kubernetes cronjobs over celery, but potato/potato

[Tansito]

haha yeah, in this case I proposed that because is what I used in the past thinking fast but if you know an alternative that simplifies it it's always welcome too

[IceKhan13]

oh, definitely not now :)

[Tansito]

Totally yes, I wanted to start the conversation because I think it's a complex one and it will require time. So absolutely for me there is a gap between take a decision and find the correct window/priority where to implement it. And for this last point, not to have the resources for that it's a correct answer to take a decision.

[akihikokuroda]

This is interesting. When this is released, we can create and watch rayjob CRD instance so that we can get events from kubernetes.

[Tansito]

This would be awesome 👀

[psschwei]

Another benefit to separating the scheduler and gateway would be easier pod name autocompletes 😆