[AIRFLOW-3121] Define closed property on StreamLogWriter #3955

jinnovation · 2018-09-26T19:44:18Z

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3121
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

This PR adds the closed property to the StreamLogWriter class—a stub function that simply returns False for compatibility reasons.

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

This PR adds a straightforward property.

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

Passes git diff upstream/master -u -- "*.py" | flake8 --diff

aoen · 2018-09-26T19:56:11Z

LGTM once CI passes

ashb · 2018-09-27T08:45:19Z

airflow/utils/log/logging_mixin.py

+        Returns False to indicate that the stream is not closed (as it will be
+        open for the duration of Airflow's lifecycle).
+
+        For compatibility.


For compatability with what please

@jinnovation

codecov-io · 2018-09-28T11:41:12Z

Codecov Report

Merging #3955 into master will increase coverage by 59.78%.
The diff coverage is 50%.

@@             Coverage Diff             @@
##           master    #3955       +/-   ##
===========================================
+ Coverage   15.14%   74.93%   +59.78%     
===========================================
  Files         200      200               
  Lines       15974    15976        +2     
===========================================
+ Hits         2419    11971     +9552     
+ Misses      13555     4005     -9550

Impacted Files	Coverage Δ
airflow/utils/log/logging_mixin.py	`96.29% <50%> (+43.13%)`	⬆️
airflow/exceptions.py	`100% <0%> (+2.85%)`	⬆️
airflow/utils/operator_resources.py	`86.95% <0%> (+4.34%)`	⬆️
airflow/executors/__init__.py	`55.76% <0%> (+5.76%)`	⬆️
airflow/utils/decorators.py	`91.66% <0%> (+14.58%)`	⬆️
airflow/settings.py	`81.15% <0%> (+15.21%)`	⬆️
airflow/hooks/oracle_hook.py	`15.47% <0%> (+15.47%)`	⬆️
airflow/task/task_runner/__init__.py	`63.63% <0%> (+18.18%)`	⬆️
airflow/utils/db.py	`33.6% <0%> (+18.4%)`	⬆️
airflow/__init__.py	`74.28% <0%> (+19.99%)`	⬆️
... and 151 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cd9a26...1f9f3e1. Read the comment docs.

[AIRFLOW-3121] Define closed property on StreamLogWriter

…3955) CP of 2d5b8a5

…3955) (#52) CP of 2d5b8a5

*Cherry-Pick contains [CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (twitter-forks#52) CP of 2d5b8a5

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (twitter-forks#52) CP of 2d5b8a5

* EWT-569 : Initial Commit for migrations * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 76fe7ac from 1.10.4 * CP Contains fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00 [CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (#63) * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00 CP contains [EWT-16]: Airflow fix for manual trigger during version upgrade (#13) * [EWT-16]: Airflow fix for manual trigger during version upgrade * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00 [CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (#63) CP of f757a54 * CP(55bb579) [AIRFLOW-5597] Linkify urls in task instance log (#16) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 94cdcf6 [CP] Contains [AIRFLOW-5597] Linkify urls in task instance log CP of f757a54 * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 4ce8d4c from 1.10.4 CP contains [TWTTR] Fix for rendering code on UI (#34) * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 299b4d8 from 1.10.4 CP contains [TWTR] CP from 1.10+twtr (#35) * 99ee040: CP from 1.10+twtr * 2e01c24: CP from 1.10.4 ([TWTR][AIRFLOW-4939] Fixup use of fallback kwarg in conf.getint) * 00cb4ae: [TWTR][AIRFLOW-XXXX] Cherry-pick d4a83bc and bump version (#21) * CP 51b1aee: Relax version requiremets (#24) * CP 67a4d1c: [CX-16266] Change with reference to 1a4c164 commit in open source (#25) * CP 54bd095: [TWTR][CX-17516] Queue tasks already being handled by the executor (#26) * CP 87fcc1c: [TWTR][CX-17516] Requeue tasks in the queued state (#27) * CP 98a1ca9: [AIRFLOW-6625] Explicitly log using utf-8 encoding (apache#7247) (#31) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : f7050fb CP Contains Experiment API path fix (#37) * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 8a689af from 1.10.4 CP Contains Export scheduler env variable into worker pods. (#38) * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 5875a15 from 1.10.4 Cp Contains [EWT-115][EWT-118] Initialise dag var to None and fix for DagModel.fileloc (missed in EWT-16) (#39) * [EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick a68e2b3 from 1.10.4 [CX-16591] Fix regex to work with impersonated clusters like airflow_scheduler_ddavydov (#42) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : e9642c2 [CP][EWT-128] Fetch task logs from worker pods (19ac45a) (#43) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : d5d0a07 [CP][AIRFLOW-6561][EWT-290]: Adding priority class and default resource for worker pod. (#47) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 9b58c88 [CP][EWT-302]Patch Pool.DEFAULT_POOL_NAME in BaseOperator (apache#8587) (#49) Open source commit id: b37ce29 * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 7b52a71 [CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (#52) CP of 2d5b8a5 * [EWT-361] Fix broken regex pattern for extracting dataflow job id (#51) Update the dataflow URL regex as per AIRFLOW-9323 * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 4b5b977 EWT-370: Use python3 to launch the dataflow job. (#53) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 596e24f * [EWT-450] fixing sla miss triggering duplicate alerts every minute (#56) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : b3d7fb4 [CP] Handle IntegrityErrors for trigger dagruns & add Stacktrace when DagFileProcessorManager gets killed (#57) CP of faaf179 - from master CP of 2102122 - from 1.10.12 * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : bac4acd [TWTR][EWT-472] Add lifecycle support while launching worker pods (#59) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 6162402 [TWTTR] Don't enqueue tasks again if already queued for K8sExecutor(#60) Basically reverting commit 87fcc1c and making changes specifically into the Celery Executor class only. * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 1991419 [CP][TWTR][EWT-377] Fix DagBag bug when a Dag has invalid schedule_interval (#61) CP of 5605d10 & apache#11462 * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 48be0f9 [TWTR][EWT-350] Reverting the last commit partially (#62) * [EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : d8c473e [CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (#63) CP of f757a54

[AIRFLOW-3121] Define closed property on StreamLogWriter

1f9f3e1

jinnovation changed the title ~~[AIRFLOW-3121] Define closed property on LoggingMixin~~ [AIRFLOW-3121] Define closed property on StreamLogWriter Sep 26, 2018

ashb requested changes Sep 27, 2018

View reviewed changes

Clarify docstring w.r.t. compatibility

8bd66af

aoen merged commit 2d5b8a5 into apache:master Oct 16, 2018

aliceabe pushed a commit to aliceabe/incubator-airflow that referenced this pull request Jan 3, 2019

[AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955)

e00b0fe

[AIRFLOW-3121] Define closed property on StreamLogWriter

wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019

[AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955)

1036157

[AIRFLOW-3121] Define closed property on StreamLogWriter

msumit added a commit to msumit/airflow that referenced this pull request Jun 18, 2020

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#…

6dfb1b6

…3955) CP of 2d5b8a5

msumit added a commit to twitter-forks/airflow that referenced this pull request Jun 18, 2020

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#…

7b52a71

…3955) (#52) CP of 2d5b8a5

ayushSethi22 pushed a commit to ayushSethi22/airflow that referenced this pull request Dec 21, 2020

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 7b52a71

bd62ed0

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (twitter-forks#52) CP of 2d5b8a5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-3121] Define closed property on StreamLogWriter #3955

[AIRFLOW-3121] Define closed property on StreamLogWriter #3955

jinnovation commented Sep 26, 2018 •

edited

Loading

aoen commented Sep 26, 2018

ashb Sep 27, 2018

Fokko Oct 12, 2018

codecov-io commented Sep 28, 2018

[AIRFLOW-3121] Define closed property on StreamLogWriter #3955

[AIRFLOW-3121] Define closed property on StreamLogWriter #3955

Conversation

jinnovation commented Sep 26, 2018 • edited Loading

Jira

Description

Tests

Commits

Documentation

Code Quality

aoen commented Sep 26, 2018

ashb Sep 27, 2018

Choose a reason for hiding this comment

Fokko Oct 12, 2018

Choose a reason for hiding this comment

codecov-io commented Sep 28, 2018

Codecov Report

jinnovation commented Sep 26, 2018 •

edited

Loading