Include removed providers when preparing packages (#36160)

As of recently, we have a possibility to schedule providers for removal.
They should be removed once they got released the last time - with the
removal note added to the documentation. We had a special step in the
process to generate documentation and build packages, but this is pretty
easy to forget it and miss such package when release is prepared.

We modify the tooling slightly - removed providers will be included
in release preparation even if they are techniclaly suspended and
would be excluded (if they were not removed).

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -161,17 +161,7 @@ Details about maintaining the SEMVER version are going to be discussed and imple
 [the related issue](https://github.com/apache/airflow/issues/11425)
 
 ```shell script
-breeze release-management prepare-provider-documentation [packages]
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), documentation for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-documentation removed.provider
+breeze release-management prepare-provider-documentation  --include-removed-providers [packages]
 ```
 
 This command will not only prepare documentation but will also help the release manager to review
@@ -193,7 +183,7 @@ When you want to regenerate the changes before the release and make sure all cha
 are updated, run it in non-interactive mode:
 
 ```shell script
-breeze release-management prepare-provider-documentation --answer yes [packages]
+breeze release-management prepare-provider-documentation  --include-removed-providers --answer yes [packages]
 ```
 
 NOTE!! In case you prepare provider's documentation in a branch different than main, you need to manually
@@ -202,7 +192,7 @@ For example if you try to build a `cncf.kubernetes` provider that is build from
 branch should be prepared like this:
 
 ```shell script
-breeze release-management prepare-provider-documentation \
+breeze release-management prepare-provider-documentation --include-removed-providers \
  --base-branch provider-cncf-kubernetes/v4-4 cncf.kubernetes
 ```
 
@@ -219,7 +209,7 @@ Regenerate the documentation templates by running the command with
 * Provider README file used when publishing package in PyPI
 
 ```shell script
-breeze release-management prepare-provider-documentation --reapply-templates-only
+breeze release-management prepare-provider-documentation --include-removed-providers --reapply-templates-only
 ```
 
 ## Open PR with suggested version releases
@@ -253,27 +243,16 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
 * Release candidate packages:
 
 ```shell script
-breeze release-management prepare-provider-packages --package-format both
+breeze release-management prepare-provider-packages  --include-removed-providers --package-format both
 ```
 
 if you only build few packages, run:
 
 ```shell script
-breeze release-management prepare-provider-packages --package-format both PACKAGE PACKAGE ....
+breeze release-management prepare-provider-packages  --include-removed-providers \
+--package-format both PACKAGE PACKAGE ....
 ```
 
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), package for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-packages --package-format both removed.provider
-```
-
-
 * Sign all your packages
 
 ```shell script
@@ -334,26 +313,17 @@ this will clean up dist folder before generating the packages, so you will only
 ```shell script
 rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
 
-breeze release-management prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
+breeze release-management prepare-provider-packages  --include-removed-providers \
+ --version-suffix-for-pypi rc1 --package-format both
 ```
 
 if you only build few packages, run:
 
 ```shell script
-breeze release-management prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), package for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-packages --package-format both removed.provider
+breeze release-management prepare-provider-packages --include-removed-providers \
+--version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
 ```
 
-
 * Verify the artifacts that would be uploaded:
 
 ```shell script
@@ -447,18 +417,6 @@ cd "${AIRFLOW_REPO_ROOT}"
 breeze build-docs apache-airflow-providers cncf.kubernetes sftp --clean-build
 ```
 
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), doc for the provider will not be built when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze build-docs removed.provider
-```
-
-
 - Now you can preview the documentation.
 
 ```shell script
@@ -484,7 +442,8 @@ way faster on multi-cpu machines when you are publishing multiple providers:
 ```shell script
 cd "${AIRFLOW_REPO_ROOT}"
 
-breeze release-management publish-docs apache-airflow-providers --package-filter 'apache-airflow-providers-*' \
+breeze release-management publish-docs apache-airflow-providers --include-removed-providers \
+    --package-filter 'apache-airflow-providers-*' \
     --override-versioned --run-in-parallel
 
 breeze release-management add-back-references all-providers
@@ -501,18 +460,7 @@ If you have providers as list of provider ids because you just released them you
 ```shell script
 cd "${AIRFLOW_REPO_ROOT}"
 
-breeze release-management publish-docs amazon apache.beam google ....
-breeze release-management add-back-references all-providers
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), docs for the provider will not be published when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management publish-docs removed.provider
+breeze release-management publish-docs  --include-removed-providers amazon apache.beam google ....
 breeze release-management add-back-references all-providers
 ```
 
@@ -712,7 +660,8 @@ rm -rf dist/*
 4) Build the packages using checked out sources
 
 ```shell
-breeze release-management prepare-provider-packages --package-format both
+breeze release-management prepare-provider-packages --include-removed-providers \
+--package-format both --include-removed-providers
 ```
 
 5) Switch to the folder where you checked out the SVN dev files

dev/breeze/src/airflow_breeze/commands/common_options.py

@@ -73,7 +73,9 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     nargs=-1,
     required=False,
     type=NotVerifiedBetterChoice(
-        get_available_packages(include_non_provider_doc_packages=True, include_all_providers=True)
+        get_available_packages(
+            include_non_provider_doc_packages=True, include_all_providers=True, include_removed=True
+        )
     ),
 )
 option_airflow_extras = click.option(
@@ -181,6 +183,12 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     envvar="PYTHON_VERSION",
     help="Python version to update sbom from. (defaults to all historical python versions)",
 )
+option_include_removed_providers = click.option(
+    "--include-removed-providers",
+    help="Whether to include providers that are removed.",
+    is_flag=True,
+    envvar="INCLUDE_REMOVED_PROVIDERS",
+)
 option_include_success_outputs = click.option(
     "--include-success-outputs",
     help="Whether to include outputs of successful parallel runs (skipped by default).",

dev/breeze/src/airflow_breeze/commands/developer_commands.py

@@ -44,6 +44,7 @@
     option_forward_credentials,
     option_github_repository,
     option_image_tag_for_running,
+    option_include_removed_providers,
     option_installation_package_format,
     option_integration,
     option_max_time,
@@ -579,41 +580,43 @@ def start_airflow(
 
 
 @main.command(name="build-docs")
-@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
-@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
 @option_builder
-@click.option(
-    "--package-filter",
-    help="List of packages to consider. You can use the full names like apache-airflow-providers-<provider>, "
-    "the short hand names or the glob pattern matching the full package name. "
-    "The list of short hand names can be found in --help output",
-    type=str,
-    multiple=True,
-)
 @click.option(
     "--clean-build",
     help="Clean inventories of Inter-Sphinx documentation and generated APIs and sphinx artifacts "
     "before the build - useful for a clean build.",
     is_flag=True,
 )
+@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
+@option_dry_run
+@option_github_repository
+@option_include_removed_providers
 @click.option(
     "--one-pass-only",
     help="Builds documentation in one pass only. This is useful for debugging sphinx errors.",
     is_flag=True,
 )
-@argument_doc_packages
-@option_github_repository
+@click.option(
+    "--package-filter",
+    help="List of packages to consider. You can use the full names like apache-airflow-providers-<provider>, "
+    "the short hand names or the glob pattern matching the full package name. "
+    "The list of short hand names can be found in --help output",
+    type=str,
+    multiple=True,
+)
+@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
 @option_verbose
-@option_dry_run
+@argument_doc_packages
 def build_docs(
-    doc_packages: tuple[str, ...],
-    docs_only: bool,
-    spellcheck_only: bool,
     builder: str,
     clean_build: bool,
+    docs_only: bool,
+    github_repository: str,
+    include_removed_providers: bool,
     one_pass_only: bool,
     package_filter: tuple[str, ...],
-    github_repository: str,
+    spellcheck_only: bool,
+    doc_packages: tuple[str, ...],
 ):
     """
     Build documents.
@@ -636,7 +639,9 @@ def build_docs(
         docs_only=docs_only,
         spellcheck_only=spellcheck_only,
         one_pass_only=one_pass_only,
-        short_doc_packages=expand_all_provider_packages(doc_packages),
+        short_doc_packages=expand_all_provider_packages(
+            doc_packages, include_removed=include_removed_providers
+        ),
     )
     cmd = "/opt/airflow/scripts/in_container/run_docs_build.sh " + " ".join(
         [shlex.quote(arg) for arg in doc_builder.args_doc_builder]

dev/breeze/src/airflow_breeze/commands/developer_commands_config.py

@@ -294,6 +294,7 @@
                 "--clean-build",
                 "--one-pass-only",
                 "--package-filter",
+                "--include-removed-providers",
                 "--github-repository",
                 "--builder",
             ],