[pip][design] PIP-281: Add notifyError method on PushSource #20807
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shibd
requested review from
Technoboy-,
codelipenghui,
RobertIndie and
poorbarcode
4 tasks
RobertIndie
reviewed
Jul 17, 2023
RobertIndie
approved these changes
Jul 17, 2023
|
@Anonymitaet Could you help review from a perspective of technical writer? |
Anonymitaet
reviewed
Aug 1, 2023
|
This PIP vote passed: https://lists.apache.org/thread/16hgd2hdsk09rvhm4rpro11vvg75jh8g |
Anonymitaet
reviewed
Aug 3, 2023
Anonymitaet
reviewed
Aug 3, 2023
Anonymitaet
reviewed
Aug 8, 2023
Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>
Anonymitaet
approved these changes
Aug 10, 2023
RobertIndie
added a commit
that referenced
this pull request
Sep 14, 2023
…the fatal exception (#21143) PIP: #21079 ### Motivation Currently, the connector and function cannot terminate the function instance if there are fatal exceptions thrown outside the function instance thread. The current implementation of the connector and Pulsar Function exception handler cannot handle the fatal exceptions that are thrown outside the function instance thread. For example, suppose we have a sink connector that uses its own threads to batch-sink the data to an external system. If any fatal exceptions occur in those threads, the function instance thread will not be aware of them and will not be able to terminate the connector. This will cause the connector to hang indefinitely. There is a related issue here: #9464 The same problem exists for the source connector. The source connector may also use a separate thread to fetch data from an external system. If any fatal exceptions happen in that thread, the connector will also hang forever. This issue has been observed for the Kafka source connector: #9464. We have fixed it by adding the notifyError method to the `PushSource` class in PIP-281: #20807. However, this does not solve the same problem that all source connectors face because not all connectors are implemented based on the `PushSource` class. The problem is same for the Pulsar Function. Currently, the function can't throw fatal exceptions to the function framework. We need to provide a way for the function developer to implement it. We need a way for the connector and function developers to throw fatal exceptions outside the function instance thread. The function framework should catch these exceptions and terminate the function accordingly. ### Modifications Introduce a new method `fatal` to the context. All the connector implementation code and the function code can use this context and call the `fatal` method to terminate the instance while raising a fatal exception. After the connector or function raises the fatal exception, the function instance thread will be interrupted. The function framework then could catch the exception, log it, and then terminate the function instance.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Refer to
pip-281.mdBTW: This is a very simple change and is forward compatible. Sorry, I didn't notice that this change requires PIP before, so the related PRs have been merged.
If this PIP vote does not pass, I revert this PR after that.
Documentation
docdoc-requireddoc-not-neededdoc-complete