Include removed providers when preparing packages

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -161,17 +161,7 @@ Details about maintaining the SEMVER version are going to be discussed and imple
 [the related issue](https://github.com/apache/airflow/issues/11425)
 
 ```shell script
-breeze release-management prepare-provider-documentation [packages]
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), documentation for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-documentation removed.provider
+breeze release-management prepare-provider-documentation  --include-removed-providers [packages]
 ```
 
 This command will not only prepare documentation but will also help the release manager to review
@@ -193,7 +183,7 @@ When you want to regenerate the changes before the release and make sure all cha
 are updated, run it in non-interactive mode:
 
 ```shell script
-breeze release-management prepare-provider-documentation --answer yes [packages]
+breeze release-management prepare-provider-documentation  --include-removed-providers --answer yes [packages]
 ```
 
 NOTE!! In case you prepare provider's documentation in a branch different than main, you need to manually
@@ -202,7 +192,7 @@ For example if you try to build a `cncf.kubernetes` provider that is build from
 branch should be prepared like this:
 
 ```shell script
-breeze release-management prepare-provider-documentation \
+breeze release-management prepare-provider-documentation --include-removed-providers \
  --base-branch provider-cncf-kubernetes/v4-4 cncf.kubernetes
 ```
 
@@ -219,7 +209,7 @@ Regenerate the documentation templates by running the command with
 * Provider README file used when publishing package in PyPI
 
 ```shell script
-breeze release-management prepare-provider-documentation --reapply-templates-only
+breeze release-management prepare-provider-documentation --include-removed-providers --reapply-templates-only
 ```
 
 ## Open PR with suggested version releases
@@ -253,27 +243,16 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
 * Release candidate packages:
 
 ```shell script
-breeze release-management prepare-provider-packages --package-format both
+breeze release-management prepare-provider-packages  --include-removed-providers --package-format both
 ```
 
 if you only build few packages, run:
 
 ```shell script
-breeze release-management prepare-provider-packages --package-format both PACKAGE PACKAGE ....
+breeze release-management prepare-provider-packages  --include-removed-providers \
+--package-format both PACKAGE PACKAGE ....
 ```
 
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), package for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-packages --package-format both removed.provider
-```
-
-
 * Sign all your packages
 
 ```shell script
@@ -334,26 +313,17 @@ this will clean up dist folder before generating the packages, so you will only
 ```shell script
 rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
 
-breeze release-management prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
+breeze release-management prepare-provider-packages  --include-removed-providers \
+ --version-suffix-for-pypi rc1 --package-format both
 ```
 
 if you only build few packages, run:
 
 ```shell script
-breeze release-management prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), package for the provider will not be prepared when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management prepare-provider-packages --package-format both removed.provider
+breeze release-management prepare-provider-packages --include-removed-providers \
+--version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
 ```
 
-
 * Verify the artifacts that would be uploaded:
 
 ```shell script
@@ -447,18 +417,6 @@ cd "${AIRFLOW_REPO_ROOT}"
 breeze build-docs apache-airflow-providers cncf.kubernetes sftp --clean-build
 ```
 
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), doc for the provider will not be built when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze build-docs removed.provider
-```
-
-
 - Now you can preview the documentation.
 
 ```shell script
@@ -484,7 +442,8 @@ way faster on multi-cpu machines when you are publishing multiple providers:
 ```shell script
 cd "${AIRFLOW_REPO_ROOT}"
 
-breeze release-management publish-docs apache-airflow-providers --package-filter 'apache-airflow-providers-*' \
+breeze release-management publish-docs apache-airflow-providers --include-removed-providers \
+    --package-filter 'apache-airflow-providers-*' \
     --override-versioned --run-in-parallel
 
 breeze release-management add-back-references all-providers
@@ -501,18 +460,7 @@ If you have providers as list of provider ids because you just released them you
 ```shell script
 cd "${AIRFLOW_REPO_ROOT}"
 
-breeze release-management publish-docs amazon apache.beam google ....
-breeze release-management add-back-references all-providers
-```
-
-NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the
-provider), docs for the provider will not be published when you prepare documentation for
-all providers - you have to specifically use the provider name in a separate command.
-For example to prepare documentation for `removed.provider` provider marked for removal you need to run
-separately this command:
-
-```shell script
-breeze release-management publish-docs removed.provider
+breeze release-management publish-docs  --include-removed-providers amazon apache.beam google ....
 breeze release-management add-back-references all-providers
 ```
 
@@ -712,7 +660,8 @@ rm -rf dist/*
 4) Build the packages using checked out sources
 
 ```shell
-breeze release-management prepare-provider-packages --package-format both
+breeze release-management prepare-provider-packages --include-removed-providers \
+--package-format both --include-removed-providers
 ```
 
 5) Switch to the folder where you checked out the SVN dev files

dev/breeze/src/airflow_breeze/commands/common_options.py

@@ -73,7 +73,9 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     nargs=-1,
     required=False,
     type=NotVerifiedBetterChoice(
-        get_available_packages(include_non_provider_doc_packages=True, include_all_providers=True)
+        get_available_packages(
+            include_non_provider_doc_packages=True, include_all_providers=True, include_removed=True
+        )
     ),
 )
 option_airflow_extras = click.option(
@@ -181,6 +183,12 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     envvar="PYTHON_VERSION",
     help="Python version to update sbom from. (defaults to all historical python versions)",
 )
+option_include_removed_providers = click.option(
+    "--include-removed-providers",
+    help="Whether to include providers that are removed.",
+    is_flag=True,
+    envvar="INCLUDE_REMOVED_PROVIDERS",
+)
 option_include_success_outputs = click.option(
     "--include-success-outputs",
     help="Whether to include outputs of successful parallel runs (skipped by default).",

dev/breeze/src/airflow_breeze/commands/developer_commands.py

@@ -44,6 +44,7 @@
     option_forward_credentials,
     option_github_repository,
     option_image_tag_for_running,
+    option_include_removed_providers,
     option_installation_package_format,
     option_integration,
     option_max_time,
@@ -579,41 +580,43 @@ def start_airflow(
 
 
 @main.command(name="build-docs")
-@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
-@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
 @option_builder
-@click.option(
-    "--package-filter",
-    help="List of packages to consider. You can use the full names like apache-airflow-providers-<provider>, "
-    "the short hand names or the glob pattern matching the full package name. "
-    "The list of short hand names can be found in --help output",
-    type=str,
-    multiple=True,
-)
 @click.option(
     "--clean-build",
     help="Clean inventories of Inter-Sphinx documentation and generated APIs and sphinx artifacts "
     "before the build - useful for a clean build.",
     is_flag=True,
 )
+@click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
+@option_dry_run
+@option_github_repository
+@option_include_removed_providers
 @click.option(
     "--one-pass-only",
     help="Builds documentation in one pass only. This is useful for debugging sphinx errors.",
     is_flag=True,
 )
-@argument_doc_packages
-@option_github_repository
+@click.option(
+    "--package-filter",
+    help="List of packages to consider. You can use the full names like apache-airflow-providers-<provider>, "
+    "the short hand names or the glob pattern matching the full package name. "
+    "The list of short hand names can be found in --help output",
+    type=str,
+    multiple=True,
+)
+@click.option("-s", "--spellcheck-only", help="Only run spell checking.", is_flag=True)
 @option_verbose
-@option_dry_run
+@argument_doc_packages
 def build_docs(
-    doc_packages: tuple[str, ...],
-    docs_only: bool,
-    spellcheck_only: bool,
     builder: str,
     clean_build: bool,
+    docs_only: bool,
+    github_repository: str,
+    include_removed_providers: bool,
     one_pass_only: bool,
     package_filter: tuple[str, ...],
-    github_repository: str,
+    spellcheck_only: bool,
+    doc_packages: tuple[str, ...],
 ):
     """
     Build documents.
@@ -636,7 +639,9 @@ def build_docs(
         docs_only=docs_only,
         spellcheck_only=spellcheck_only,
         one_pass_only=one_pass_only,
-        short_doc_packages=expand_all_provider_packages(doc_packages),
+        short_doc_packages=expand_all_provider_packages(
+            doc_packages, include_removed=include_removed_providers
+        ),
     )
     cmd = "/opt/airflow/scripts/in_container/run_docs_build.sh " + " ".join(
         [shlex.quote(arg) for arg in doc_builder.args_doc_builder]

dev/breeze/src/airflow_breeze/commands/developer_commands_config.py

@@ -294,6 +294,7 @@
                 "--clean-build",
                 "--one-pass-only",
                 "--package-filter",
+                "--include-removed-providers",
                 "--github-repository",
                 "--builder",
             ],