From a98ca003533902e7144911c270751e3b7f1c5fff Mon Sep 17 00:00:00 2001
From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Thu, 3 Oct 2024 17:56:46 +0100
Subject: [PATCH 1/6] Add hybrid workflow documentation for Azure

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/azure.md | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/docs/azure.md b/docs/azure.md
index c17c9926a6..3e8f2b0dc8 100644
--- a/docs/azure.md
+++ b/docs/azure.md
@@ -405,6 +405,47 @@ The value of the setting must be the identifier of a subnet available in the vir
 Batch Authentication with Shared Keys does not allow to link external resources (like Virtual Networks) to the pool. Therefore, Active Directory Authentication must be used in conjunction with the `virtualNetwork` setting.
 :::
 
+### Hybrid workloads
+
+Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to Azure Batch.
+
+To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the Azure Batch configuration to the subset of processes that you want to offload. For example:
+
+```groovy
+process {
+    withLabel: bigTask {
+        executor = 'azurebatch'
+        queue = 'my-batch-pool'
+        container = 'my/image:tag'
+    }
+}
+
+azure {
+    storage {
+        accountName = '<YOUR STORAGE ACCOUNT NAME>'
+        accountKey = '<YOUR STORAGE ACCOUNT KEY>'
+    }
+    batch {
+        location = '<YOUR LOCATION>'
+        accountName = '<YOUR BATCH ACCOUNT NAME>'
+        accountKey = '<YOUR BATCH ACCOUNT KEY>'
+    }
+}
+```
+
+With the above configuration, processes with the `bigTask` {ref}`process-label` will run on Azure Batch, while the remaining processes will run in the local computer.
+
+Then launch the pipeline with the `-bucket-dir` option to specify an Azure Blob Storage path for the jobs computed with Azure Batch and, optionally, the `-work-dir` to specify the local storage for the jobs computed locally:
+
+```bash
+nextflow run <script or project name> -bucket-dir az://my-container/some/path
+```
+
+:::{warning}
+The Azure Blob Storage path needs to contain at least one sub-directory (e.g. `az://my-container/work` rather than `az://my-container`).
+:::
+
+
 ## Microsoft Entra
 
 Using Microsoft Entra for role-based access control is more secure than using access keys and should be used wherever possible. You can authenticate to Azure Entra using a Managed Identity when running on resources within the Azure environment, or by authenticating as an Azure Service Principal when running on external resources.

From 51f384c7b53edb95beadff8a878ff0946b5dda9f Mon Sep 17 00:00:00 2001
From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Thu, 3 Oct 2024 18:09:38 +0100
Subject: [PATCH 2/6] Harmonise hybrid execution documentation

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/aws.md    | 44 +++++++++++++++++++++-----------------------
 docs/azure.md  |  5 +++--
 docs/google.md | 15 +++++++++------
 3 files changed, 33 insertions(+), 31 deletions(-)

diff --git a/docs/aws.md b/docs/aws.md
index 4763f05c67..121806a2de 100644
--- a/docs/aws.md
+++ b/docs/aws.md
@@ -408,34 +408,13 @@ To do that, first create a **Job Definition** in the AWS Console (or by other me
 process.container = 'job-definition://your-job-definition-name'
 ```
 
-### Pipeline execution
-
-The pipeline can be launched either in a local computer or an EC2 instance. The latter is suggested for heavy or long-running workloads.
-
-Pipeline input data can be stored either locally or in an [S3](https://aws.amazon.com/s3/) bucket. The pipeline execution must specify an S3 bucket to store intermediate results with the `-bucket-dir` (`-b`) command line option. For example:
-
-```bash
-nextflow run my-pipeline -bucket-dir s3://my-bucket/some/path
-```
-
-:::{warning}
-The bucket path should include at least a top level directory name, e.g. `s3://my-bucket/work` rather than `s3://my-bucket`.
-:::
-
 ### Hybrid workloads
 
 Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to AWS Batch.
 
-To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example:
 
 ```groovy
-aws {
-    region = 'eu-west-1'
-    batch {
-        cliPath = '/home/ec2-user/miniconda/bin/aws'
-    }
-}
-
 process {
     withLabel: bigTask {
         executor = 'awsbatch'
@@ -443,9 +422,28 @@ process {
         container = 'my/image:tag'
     }
 }
+
+aws {
+    region = 'eu-west-1'
+    batch {
+        cliPath = '/home/ec2-user/miniconda/bin/aws'
+    }
+}
 ```
 
-With the above configuration, processes with the `bigTask` {ref}`process-label` will run on AWS Batch, while the remaining processes with run in the local computer.
+With the above configuration, processes with the `bigTask` {ref}`process-label` will run on AWS Batch, while the remaining processes will run in the local computer.
+
+Then launch the pipeline with the -bucket-dir option to specify an AWS S3 path for the jobs computed with AWS Batch and, optionally, the -work-dir to specify the local storage for the jobs computed locally:
+
+```bash
+nextflow run <script or project name> -bucket-dir s3://my-bucket/some/path
+```
+
+:::warning
+The AWS S3 path needs to contain at least one sub-directory (e.g. `s3://my-bucket/work` rather than `s3://my-bucket`).
+:::
+
+Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
 
 ### Volume mounts
 
diff --git a/docs/azure.md b/docs/azure.md
index 3e8f2b0dc8..f063db4cb4 100644
--- a/docs/azure.md
+++ b/docs/azure.md
@@ -409,7 +409,7 @@ Batch Authentication with Shared Keys does not allow to link external resources
 
 Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to Azure Batch.
 
-To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the Azure Batch configuration to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration to apply the Azure Batch configuration to the subset of processes that you want to offload. For example:
 
 ```groovy
 process {
@@ -433,7 +433,7 @@ azure {
 }
 ```
 
-With the above configuration, processes with the `bigTask` {ref}`process-label` will run on Azure Batch, while the remaining processes will run in the local computer.
+With the above configuration, processes with the bigTask process-label will run on Azure Batch, while the remaining processes will run in the local computer.
 
 Then launch the pipeline with the `-bucket-dir` option to specify an Azure Blob Storage path for the jobs computed with Azure Batch and, optionally, the `-work-dir` to specify the local storage for the jobs computed locally:
 
@@ -445,6 +445,7 @@ nextflow run <script or project name> -bucket-dir az://my-container/some/path
 The Azure Blob Storage path needs to contain at least one sub-directory (e.g. `az://my-container/work` rather than `az://my-container`).
 :::
 
+Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
 
 ## Microsoft Entra
 
diff --git a/docs/google.md b/docs/google.md
index 30bd8d4f84..136b24c44b 100644
--- a/docs/google.md
+++ b/docs/google.md
@@ -398,34 +398,37 @@ For an exhaustive list of error codes, refer to the official Google Life Science
 
 ### Hybrid execution
 
-Nextflow allows the use of multiple executors in the same workflow. This feature enables the deployment of hybrid workloads, in which some jobs are executed in the local computer or local computing cluster, and some jobs are offloaded to Google Life Sciences.
+Nextflow allows the use of multiple executors in the same workflow. This feature enables the deployment of hybrid workloads, in which some jobs are executed in the local computer or local computing cluster, and some jobs are offloaded to Google Cloud (either Google Batch or Google Life Sciences).
 
-To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration file to apply the Google Life Sciences executor to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration file to apply the Google Cloud executor to the subset of processes that you want to offload. For example:
 
 ```groovy
 process {
     withLabel: bigTask {
-        executor = 'google-lifesciences'
+        executor = 'google-batch' // or 'google-lifesciences'
         container = 'my/image:tag'
     }
 }
 
 google {
     project = 'your-project-id'
-    zone = 'europe-west1-b'
+    location = 'us-central1' // for Google Batch
+    // zone = 'us-central1-a' // for Google Life Sciences
 }
 ```
 
-Then launch the pipeline with the `-bucket-dir` option to specify a Google Storage path for the jobs computed with Google Life Sciences and, optionally, the `-work-dir` to specify the local storage for the jobs computed locally:
+Then launch the pipeline with the `-bucket-dir` option to specify a Google Storage path for the jobs computed with Google Cloud and, optionally, the `-work-dir` to specify the local storage for the jobs computed locally:
 
 ```bash
 nextflow run <script or project name> -bucket-dir gs://my-bucket/some/path
 ```
 
-:::{warning}
+:::warning
 The Google Storage path needs to contain at least one sub-directory (e.g. `gs://my-bucket/work` rather than `gs://my-bucket`).
 :::
 
+Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
+
 ### Limitations
 
 - Compute resources in Google Cloud are subject to [resource quotas](https://cloud.google.com/compute/quotas), which may affect your ability to run pipelines at scale. You can request quota increases, and your quotas may automatically increase over time as you use the platform. In particular, GPU quotas are initially set to 0, so you must explicitly request a quota increase in order to use GPUs. You can initially request an increase to 1 GPU at a time, and after one billing cycle you may be able to increase it further.

From aa57efe13441c7396f221d6302a8884d308a6555 Mon Sep 17 00:00:00 2001
From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Thu, 3 Oct 2024 18:12:18 +0100
Subject: [PATCH 3/6] fixup warning

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/aws.md    | 2 +-
 docs/google.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/aws.md b/docs/aws.md
index 121806a2de..81211e42ea 100644
--- a/docs/aws.md
+++ b/docs/aws.md
@@ -439,7 +439,7 @@ Then launch the pipeline with the -bucket-dir option to specify an AWS S3 path f
 nextflow run <script or project name> -bucket-dir s3://my-bucket/some/path
 ```
 
-:::warning
+:::{warning}
 The AWS S3 path needs to contain at least one sub-directory (e.g. `s3://my-bucket/work` rather than `s3://my-bucket`).
 :::
 
diff --git a/docs/google.md b/docs/google.md
index 136b24c44b..7f4e9c7033 100644
--- a/docs/google.md
+++ b/docs/google.md
@@ -423,7 +423,7 @@ Then launch the pipeline with the `-bucket-dir` option to specify a Google Stora
 nextflow run <script or project name> -bucket-dir gs://my-bucket/some/path
 ```
 
-:::warning
+:::{warning}
 The Google Storage path needs to contain at least one sub-directory (e.g. `gs://my-bucket/work` rather than `gs://my-bucket`).
 :::
 

From f25a8fdd4386408df3ff816b4d69045f8f3ab38e Mon Sep 17 00:00:00 2001
From: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Mon, 7 Oct 2024 09:43:14 +0100
Subject: [PATCH 4/6] Apply suggestions from code review

Co-authored-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/aws.md    | 6 ++++--
 docs/azure.md  | 6 ++++--
 docs/google.md | 6 ++++--
 3 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/docs/aws.md b/docs/aws.md
index 81211e42ea..ccdf3d7180 100644
--- a/docs/aws.md
+++ b/docs/aws.md
@@ -412,7 +412,7 @@ process.container = 'job-definition://your-job-definition-name'
 
 Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to AWS Batch.
 
-To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example:
 
 ```groovy
 process {
@@ -443,7 +443,9 @@ nextflow run <script or project name> -bucket-dir s3://my-bucket/some/path
 The AWS S3 path needs to contain at least one sub-directory (e.g. `s3://my-bucket/work` rather than `s3://my-bucket`).
 :::
 
-Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
+:::{note}
+Nextflow will automatically manage the transfer of input and output files between the local and cloud environments when using hybrid workloads.
+:::
 
 ### Volume mounts
 
diff --git a/docs/azure.md b/docs/azure.md
index f063db4cb4..4cfe7e24fa 100644
--- a/docs/azure.md
+++ b/docs/azure.md
@@ -409,7 +409,7 @@ Batch Authentication with Shared Keys does not allow to link external resources
 
 Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to Azure Batch.
 
-To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration to apply the Azure Batch configuration to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the Azure Batch configuration to the subset of processes that you want to offload. For example:
 
 ```groovy
 process {
@@ -445,7 +445,9 @@ nextflow run <script or project name> -bucket-dir az://my-container/some/path
 The Azure Blob Storage path needs to contain at least one sub-directory (e.g. `az://my-container/work` rather than `az://my-container`).
 :::
 
-Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
+:::{note}
+Nextflow will automatically manage the transfer of input and output files between the local and cloud environments when using hybrid workloads.
+:::
 
 ## Microsoft Entra
 
diff --git a/docs/google.md b/docs/google.md
index 7f4e9c7033..b2a0bdfa98 100644
--- a/docs/google.md
+++ b/docs/google.md
@@ -400,7 +400,7 @@ For an exhaustive list of error codes, refer to the official Google Life Science
 
 Nextflow allows the use of multiple executors in the same workflow. This feature enables the deployment of hybrid workloads, in which some jobs are executed in the local computer or local computing cluster, and some jobs are offloaded to Google Cloud (either Google Batch or Google Life Sciences).
 
-To enable this feature, use one or more `config-process-selectors` in your Nextflow configuration file to apply the Google Cloud executor to the subset of processes that you want to offload. For example:
+To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration file to apply the Google Cloud executor to the subset of processes that you want to offload. For example:
 
 ```groovy
 process {
@@ -427,7 +427,9 @@ nextflow run <script or project name> -bucket-dir gs://my-bucket/some/path
 The Google Storage path needs to contain at least one sub-directory (e.g. `gs://my-bucket/work` rather than `gs://my-bucket`).
 :::
 
-Note that when using hybrid workloads, Nextflow will automatically manage the transfer of input and output files between the local and cloud environments as needed.
+:::{note}
+Nextflow will automatically manage the transfer of input and output files between the local and cloud environments when using hybrid workloads.
+:::
 
 ### Limitations
 

From 0f62fa5b00b47af0d9df43946f258e3c4104c9a5 Mon Sep 17 00:00:00 2001
From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Fri, 18 Oct 2024 14:20:45 +0100
Subject: [PATCH 5/6] fixup post merge

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/azure.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/azure.md b/docs/azure.md
index 4399998a49..df63c5e816 100644
--- a/docs/azure.md
+++ b/docs/azure.md
@@ -433,9 +433,9 @@ azure {
 }
 ```
 
-With the above configuration, processes with the bigTask {ref}`process-label` run on Azure Batch, while the remaining processes run on the local computer.
+With the above configuration, processes with the bigTask {ref}`process-label` will run on Azure Batch, while the remaining processes will run on the local computer.
 
-Next, launch the pipeline with the `-bucket-dir` option to specify an Azure Blob Storage path for the jobs running on Azure Batch, and optionally, use the `-work-dir` option to specify local storage for the jobs running locally:
+Then launch the pipeline with the `-bucket-dir` option to specify an Azure Blob Storage path for the jobs computed with Azure Batch, and optionally, use the `-work-dir` option to specify the local storage for the jobs computed locally:
 
 ```bash
 nextflow run <script or project name> -bucket-dir az://my-container/some/path

From 19bca6f1ebe6a9fdd2fe45167fdea96d9fb64f48 Mon Sep 17 00:00:00 2001
From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Date: Fri, 18 Oct 2024 14:41:42 +0100
Subject: [PATCH 6/6] Drop miscellaneous AWS Batch CLI path

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
---
 docs/aws.md | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/docs/aws.md b/docs/aws.md
index ccdf3d7180..5c48cb56af 100644
--- a/docs/aws.md
+++ b/docs/aws.md
@@ -425,9 +425,6 @@ process {
 
 aws {
     region = 'eu-west-1'
-    batch {
-        cliPath = '/home/ec2-user/miniconda/bin/aws'
-    }
 }
 ```