Officially support HA for scheduler component (AIP-15)

[ashb]

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651

• Detect orphaned task instances by SchedulerJob id and heartbeat #10729 Detect orphaned task instances by SchedulerJob id and heartbeat
• Mark task as failed when it fails "sending" in Celery #10881 Mark task as failed when it fails sending in Celery
• Allow CeleryExecutor to "adopt" an orphaned queued or running task #10949 Allow CeleryExecutor to "adopt" an orphaned queued or running task
• Kubernetes executor can adopt tasks from other schedulers #10996 Allow Kubernetes executor to adopt tasks
• Add task adoption to CeleryKubernetesExecutor #11244 Add adoption support in to the new CeleryKubernetesExecutor (added in Add CeleryKubernetesExecutor #10901)
• The big one - scheduler HA pr Fully support running more than one scheduler concurrently #10956
• Perform "mini scheduling run" after task has finished #11498 Add "fast follow" to end of task run (this is a mini "scheduler" run at the end of task execution.)
• Performance optimisation PRs
• Decide if we need Ensure idle database connections don't hang around for ever #10933 or not
• Ensure smart sensor DAG is loaded by scheduler even when DAG serialization is enabled.

[mik-laj]

@ashb @kaxil Could you please share some more details about the change?

In particular, I would like answers to the following questions:

• Is there a plan to implement this AIP? If so, can I have any preview?
• Do you need help with this task? If you plan to do it yourself, it's okay, but if you would like other people to contribute, I would like to know how to do it.
• How can I get more information on this subject? Slack channel, DM, Jira, Email? Each contributor has a different preference for communication.

With these questions, I would like to remove barriers for new contributors who do not follow the activity of the entire community. Hope this helps speed up your work on key features for the new Airflow 2.0.

[ashb]

Yes, I have started work on this, and got reasonably far with it (far enough to start running performance tests, but there are a few things I still have as to-do's such as SLAs or per task concurrency limits) but unfortunately I want able to get the code in a situation where I could pr it before going off on paternity leave - as soon as I'm back (September) getting this landed is my top priority.

[mik-laj]

Link: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651

[ryw]

@ashb since you're done w/ work for 2.0 on this issue, maybe close this out and create any new tickets for things you want to do post-2.0?