Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Redis task handler #31855

Merged
merged 2 commits into from
Jul 23, 2023
Merged

Add Redis task handler #31855

merged 2 commits into from
Jul 23, 2023

Commits on Jul 22, 2023

  1. Allow FileTaskHandler to delegate to instances of logging.Handler 

    FileTaskHandler is the base class for logging handlers, including those that
don't log to files via delegating to logging.FileHandler, e.g. in the
CloudwatchTaskHandler at
https://github.com/apache/airflow/blob/2940b9fa55a6a72c60c2162e541631addec3d6b8/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py#L67

It is suspected that type checking is not enabled in this part of the
CloudwatchTaskHandler otherwise it would have already been failing.

This change adjusts the base class so if type checking is enabled in the task
handler, if it delegates to a logging.Handler that is not a logging.FileHandler
as the CloudWatchHandler, then the type checking should pass.

This was originally part of apache#31855 and
split out.

related: apache#31834
    @michalc
    michalc committed Jul 22, 2023
    Configuration menu
    Copy the full SHA
    978bf0e View commit details
    Browse the repository at this point in the history

  2. Add Redis task handler 

    This stores log lines in Redis up to a configured maximum log lines, always
keeping the most recent, up to a configured TTL.

This deviates from other existing task handlers in that it accepts a connection
ID. This allows it to be used in addition to other handlers, and so allows a
graceful/reversible transition from one logging system to another.

This is particularly useful in situations that use Redis as a message broker,
where additional infrastructure isn't desired.

closes: apache#31834
    @michalc
    michalc committed Jul 22, 2023
    Configuration menu
    Copy the full SHA
    9ca7921 View commit details
    Browse the repository at this point in the history