Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let clients of streaming workunit handler consume independent workunit streams #10092

Open
gshuflin opened this issue Jun 18, 2020 · 2 comments

Comments

@gshuflin
Copy link
Contributor

We currently have a global max verbosity for workunits handled by the StreamingWorkunitHandler system. Instead, individual handlers should be able to individually request which level of workunits they want to receive.

@Eric-Arellano Eric-Arellano self-assigned this Mar 12, 2021
@Eric-Arellano
Copy link
Contributor

Related, switch each callback to run as a separate thread. The main benefit is that some callbacks could finish async, and others synchronously, whereas currently all must be the same.

However, this is blocked by:

to poll independently from multiple threads you’d need to change the rust-side polling method to keep multiple streams of workunits, or have one thread that polled and then sent copies to the two handlers or something

each call to “poll_workunits” has a sideeffect of consuming everything

@Eric-Arellano
Copy link
Contributor

See #11692 for an example of how to get the thread-per-callback working on the Python side. However, this suffers from the above problem with Rust's poll_workunits being destructive and not safe with multiple threads.

@stuhood stuhood changed the title Let clients of streaming workunit handler control level individually Let clients of streaming workunit handler consume independent workunit streams Jun 16, 2021
@Eric-Arellano Eric-Arellano removed their assignment Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants