Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run an extra command whenever the queue is empty #578

Closed
fzyzcjy opened this issue Nov 28, 2024 · 6 comments · Fixed by #583
Closed

Run an extra command whenever the queue is empty #578

fzyzcjy opened this issue Nov 28, 2024 · 6 comments · Fixed by #583

Comments

@fzyzcjy
Copy link
Contributor

fzyzcjy commented Nov 28, 2024

A detailed description of the feature you would like to see added.

Hi thanks for the tool! My (summarized) use case is like this: Submit a few deep learning jobs to my workstation, and when all are done (i.e. GPU is finally idle), send me a Slack message such that I can come and use the idle GPU. For example, I will need the idle GPU to tentatively run some new commands, debug them, and when they look good, submit it to the queue.

Thus, it would be great to have a feature that, when the job is empty, run an extra command to notify users.

EDIT: Looks like I can use the "Callback" feature to handle it. Seems need to query pueue to know whether the queue is empty now, thus it would be great if this information could be provided as a variable.

EDIT: Looks like it can be done by e.g.

parameters.insert("id", task.id.to_string());
add a line here, giving state.tasks.len().

Explain your usecase of the requested feature

(see above)

Alternatives

No response

Additional context

No response

@fzyzcjy fzyzcjy changed the title Notify when all jobs are done (i.e. server is finally idle) Run an extra command whenever the queue is empty Nov 28, 2024
@fzyzcjy
Copy link
Contributor Author

fzyzcjy commented Nov 28, 2024

(The content in the root post is edited for more details)

@Nukesor
Copy link
Owner

Nukesor commented Nov 28, 2024

I see how that can be useful.

How about, we add a simple boolean to the template variables. Something like group_finished?
I'm against just using state.tasks.len() as users will probably be interested in which group just finished. This will give them more control on what should happen based on the finished group.

The group is already part of the variables that're available.

@Nukesor
Copy link
Owner

Nukesor commented Nov 28, 2024

You could then do stuff like

if [[ "true" = "{{ group_finished }}" ]]; then
  if [[ "default" = "{{ group }}" ]]; then
    # Do your stuff
  fi
fi

@fzyzcjy
Copy link
Contributor Author

fzyzcjy commented Nov 28, 2024

we add a simple boolean to the template variables. Something like group_finished

Thanks, that looks pretty reasonable!

@Nukesor
Copy link
Owner

Nukesor commented Nov 29, 2024

One more question.

When should a queue be considered finished?
I feel at least when there're no more queued, running or paused entries. But what about stashed entries with a delay? I guess a group shouldn't be considered "finished" if there's still a task that'll be queued sometime in the future.

stashed entries without a delay oh the other hand should probably be ignored?

@fzyzcjy
Copy link
Contributor Author

fzyzcjy commented Nov 29, 2024

Hmm, then maybe provide fine-grained details, such as {{ num_queued }}, {{ num_stashed }}, etc?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants