diff --git a/README.md b/README.md index 8fb4770..d683318 100644 --- a/README.md +++ b/README.md @@ -247,31 +247,36 @@ implements a consistent runtime environment. The current providers are: - local - google-v2 (the default) -- google-cls-v2 (*new*) +- google-cls-v2 +- google-batch (*new*) More details on the runtime environment implemented by the backend providers can be found in [dsub backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md). -### Differences between `google-v2` and `google-cls-v2` +### Differences between `google-v2`, `google-cls-v2` and `google-batch` The `google-cls-v2` provider is built on the Cloud Life Sciences `v2beta` API. This API is very similar to its predecessor, the Genomics `v2alpha1` API. Details of the differences can be found in the [Migration Guide](https://cloud.google.com/life-sciences/docs/how-tos/migration). -`dsub` largely hides the differences between the two APIs, but there are a +The `google-batch` provider is built on the Cloud Batch API. +Details of Cloud Life Sciences versus Batch can be found in this +[Migration Guide](https://cloud.google.com/batch/docs/migrate-to-batch-from-cloud-life-sciences). + +`dsub` largely hides the differences between the APIs, but there are a few difference to note: -- `v2beta` is a regional service, `v2alpha1` is a global service +- `v2beta` and Cloud Batch are regional services, `v2alpha1` is a global service What this means is that with `v2alpha1`, the metadata about your tasks -(called "operations"), is stored in a global database, while with `v2beta`, the -metadata about your tasks are stored in a regional database. If your operation -information needs to stay in a particular region, use the `v2beta` API -(the `google-cls-v2` provider), and specify the `--location` where your -operation information should be stored. +(called "operations"), is stored in a global database, while with `v2beta` and +Cloud Batch, the metadata about your tasks are stored in a regional database. If +your operation/job information needs to stay in a particular region, use the +`v2beta` or Batch API (the `google-cls-v2` or `google-batch` provider), and +specify the `--location` where your operation/job information should be stored. -- The `--regions` and `--zones` flags can be omitted when using `google-cls-v2` +- The `--regions` and `--zones` flags can be omitted when using `google-cls-v2` and `google-batch` The `--regions` and `--zones` flags for `dsub` specify where the tasks should run. More specifically, this specifies what Compute Engine Zones to use for @@ -280,9 +285,9 @@ the VMs that run your tasks. With the `google-v2` provider, there is no default region or zone, and thus one of the `--regions` or `--zones` flags is required. -With `google-cls-v2`, the `--location` flag defaults to `us-central1`, and -if the `--regions` and `--zones` flags are omitted, the `location` will be -used as the default `regions` list. +With `google-cls-v2` and `google-batch`, the `--location` flag defaults to +`us-central1`, and if the `--regions` and `--zones` flags are omitted, the +`location` will be used as the default `regions` list. ## `dsub` features @@ -466,16 +471,17 @@ local directory in a similar fashion to support your local development. ##### Mounting a Google Cloud Storage bucket -To have the `google-v2` or `google-cls-v2` provider mount a Cloud Storage bucket -using [Cloud Storage FUSE](https://cloud.google.com/storage/docs/gcs-fuse), -use the `--mount` command line flag: +To have the `google-v2`, `google-cls-v2`, or `google-batch` provider mount a +Cloud Storage bucket using +[Cloud Storage FUSE](https://cloud.google.com/storage/docs/gcs-fuse), use the +`--mount` command line flag: --mount RESOURCES=gs://mybucket -The bucket will be mounted into the Docker container running your `--script` -or `--command` and the location made available via the environment variable -`${RESOURCES}`. Inside your script, you can reference the mounted path using the -environment variable. Please read +The bucket will be mounted read-only into the Docker container running your +`--script` or `--command` and the location made available via the environment +variable `${RESOURCES}`. Inside your script, you can reference the mounted path +using the environment variable. Please read [Key differences from a POSIX file system](https://cloud.google.com/storage/docs/gcs-fuse#notes) and [Semantics](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md) before using Cloud Storage FUSE. diff --git a/dsub/_dsub_version.py b/dsub/_dsub_version.py index 3d8de01..1cbabaa 100644 --- a/dsub/_dsub_version.py +++ b/dsub/_dsub_version.py @@ -26,4 +26,4 @@ 0.1.3.dev0 -> 0.1.3 -> 0.1.4.dev0 -> ... """ -DSUB_VERSION = '0.4.11' +DSUB_VERSION = '0.4.12' diff --git a/dsub/providers/batch_dummy.py b/dsub/providers/batch_dummy.py index 733593c..ad37125 100644 --- a/dsub/providers/batch_dummy.py +++ b/dsub/providers/batch_dummy.py @@ -39,6 +39,7 @@ class AllocationPolicy(object): Disk = None NetworkPolicy = None Accelerator = None + LocationPolicy = None class LogsPolicy(object): Destination = None diff --git a/dsub/providers/google_batch.py b/dsub/providers/google_batch.py index e6b6c4a..8ec02a5 100644 --- a/dsub/providers/google_batch.py +++ b/dsub/providers/google_batch.py @@ -41,6 +41,8 @@ from . import batch_dummy as batch_v1 # pylint: enable=g-import-not-at-top _PROVIDER_NAME = 'google-batch' +# Index of the prepare action in the runnable list +_PREPARE_INDEX = 1 # Create file provider whitelist. _SUPPORTED_FILE_PROVIDERS = frozenset([job_model.P_GCS]) @@ -210,9 +212,8 @@ def raw_task_data(self): def _try_op_to_job_descriptor(self): # The _META_YAML_REPR field in the 'prepare' action enables reconstructing # the original job descriptor. - # TODO: Currently, we set the environment across all runnables - # We really only want the env for the prepare action (runnable) here. - env = google_batch_operations.get_environment(self._op) + # We only need the env for the prepare action (runnable) here. + env = google_batch_operations.get_environment(self._op, _PREPARE_INDEX) if not env: return @@ -328,9 +329,8 @@ def get_field(self, field: str, default: str = None): return value if value else default def _try_op_to_script_body(self): - # TODO: Currently, we set the environment across all runnables - # We really only want the env for the prepare action (runnable) here. - env = google_batch_operations.get_environment(self._op) + # We only need the env for the prepare action (runnable) here. + env = google_batch_operations.get_environment(self._op, _PREPARE_INDEX) if env: return ast.literal_eval(env.get(google_utils.SCRIPT_VARNAME)) @@ -404,6 +404,28 @@ def _batch_handler_def(self): def _operations_cancel_api_def(self): return batch_v1.BatchServiceClient().delete_job + def _get_batch_job_regions(self, regions, zones) -> List[str]: + """Returns the list of regions and zones to use for a Batch Job request. + + If neither regions nor zones were specified for the Job, then use the + Batch Job API location as the default region. + + Regions need to be prefixed with "regions/" and zones need to be prefixed + with "zones/" as documented in + https://cloud.google.com/batch/docs/reference/rest/v1/projects.locations.jobs#LocationPolicy + + Args: + regions (str): A space separated list of regions to use for the Job. + zones (str): A space separated list of zones to use for the Job. + """ + if regions: + regions = [f'regions/{region}' for region in regions] + if zones: + zones = [f'zones/{zone}' for zone in zones] + if not regions and not zones: + return [f'regions/{self._location}'] + return (regions or []) + (zones or []) + def _get_create_time_filters(self, create_time_min, create_time_max): # TODO: Currently, Batch API does not support filtering by create t. return [] @@ -425,11 +447,48 @@ def _get_logging_env(self, logging_uri, user_project, include_filter_script): return env + def _format_batch_job_id(self, task_metadata, job_metadata) -> str: + # Each dsub task is submitted as its own Batch API job, so we + # append the dsub task-id and task-attempt to the job-id for the + # batch job ID. + # For single-task dsub jobs, there is no task-id, so use 0. + # Use a '-' character as the delimeter because Batch API job ID + # must match regex ^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$ + task_id = task_metadata.get('task-id') or 0 + task_attempt = task_metadata.get('task-attempt') or 0 + batch_job_id = job_metadata.get('job-id') + return f'{batch_job_id}-{task_id}-{task_attempt}' + + def _get_gcs_volumes(self, mounts) -> List[batch_v1.types.Volume]: + # Return a list of GCS volumes for the Batch Job request. + gcs_volumes = [] + for gcs_mount in param_util.get_gcs_mounts(mounts): + mount_path = os.path.join(_VOLUME_MOUNT_POINT, gcs_mount.docker_path) + # Normalize mount path because API does not allow trailing slashes + normalized_mount_path = os.path.normpath(mount_path) + gcs_volume = google_batch_operations.build_gcs_volume( + gcs_mount.value[len('gs://') :], normalized_mount_path, ['-o ro'] + ) + gcs_volumes.append(gcs_volume) + return gcs_volumes + + def _get_gcs_volumes_for_user_command(self, mounts) -> List[str]: + # Return a list of GCS volumes to be included with the + # user-command runnable + user_command_volumes = [] + for gcs_mount in param_util.get_gcs_mounts(mounts): + volume_mount_point = os.path.normpath( + os.path.join(_VOLUME_MOUNT_POINT, gcs_mount.docker_path) + ) + data_mount_point = os.path.normpath( + os.path.join(_DATA_MOUNT_POINT, gcs_mount.docker_path) + ) + user_command_volumes.append(f'{volume_mount_point}:{data_mount_point}') + return user_command_volumes + def _create_batch_request( self, task_view: job_model.JobDescriptor, - job_id, - all_envs: List[batch_v1.types.Environment], ): job_metadata = task_view.job_metadata job_params = task_view.job_params @@ -516,6 +575,29 @@ def _create_batch_request( ) ) + envs = job_params['envs'] | task_params['envs'] + inputs = job_params['inputs'] | task_params['inputs'] + outputs = job_params['outputs'] | task_params['outputs'] + mounts = job_params['mounts'] + gcs_volumes = self._get_gcs_volumes(mounts) + + prepare_env = google_batch_operations.build_environment( + self._get_prepare_env( + script, task_view, inputs, outputs, mounts, _DATA_MOUNT_POINT + ) + ) + localization_env = google_batch_operations.build_environment( + self._get_localization_env(inputs, user_project, _DATA_MOUNT_POINT) + ) + user_environment = google_batch_operations.build_environment( + self._build_user_environment( + envs, inputs, outputs, mounts, _DATA_MOUNT_POINT + ) + ) + delocalization_env = google_batch_operations.build_environment( + self._get_delocalization_env(outputs, user_project, _DATA_MOUNT_POINT) + ) + # Build the list of runnables (aka actions) runnables = [] @@ -538,7 +620,7 @@ def _create_batch_request( run_in_background=False, always_run=False, image_uri=google_utils.CLOUD_SDK_IMAGE, - environment=None, + environment=prepare_env, entrypoint='/bin/bash', volumes=[f'{_VOLUME_MOUNT_POINT}:{_DATA_MOUNT_POINT}'], commands=['-c', prepare_command], @@ -551,7 +633,7 @@ def _create_batch_request( run_in_background=False, always_run=False, image_uri=google_utils.CLOUD_SDK_IMAGE, - environment=None, + environment=localization_env, entrypoint='/bin/bash', volumes=[f'{_VOLUME_MOUNT_POINT}:{_DATA_MOUNT_POINT}'], commands=[ @@ -566,15 +648,18 @@ def _create_batch_request( ) ) + user_command_volumes = [f'{_VOLUME_MOUNT_POINT}:{_DATA_MOUNT_POINT}'] + for gcs_volume in self._get_gcs_volumes_for_user_command(mounts): + user_command_volumes.append(gcs_volume) runnables.append( # user-command google_batch_operations.build_runnable( run_in_background=False, always_run=False, image_uri=job_resources.image, - environment=None, + environment=user_environment, entrypoint='/usr/bin/env', - volumes=[f'{_VOLUME_MOUNT_POINT}:{_DATA_MOUNT_POINT}'], + volumes=user_command_volumes, commands=[ 'bash', '-c', @@ -593,7 +678,7 @@ def _create_batch_request( run_in_background=False, always_run=False, image_uri=google_utils.CLOUD_SDK_IMAGE, - environment=None, + environment=delocalization_env, entrypoint='/bin/bash', volumes=[f'{_VOLUME_MOUNT_POINT}:{_DATA_MOUNT_POINT}:ro'], commands=[ @@ -681,10 +766,17 @@ def _create_batch_request( no_external_ip_address=job_resources.use_private_address, ) + location_policy = google_batch_operations.build_location_policy( + allowed_locations=self._get_batch_job_regions( + regions=job_resources.regions, zones=job_resources.zones + ), + ) + allocation_policy = google_batch_operations.build_allocation_policy( ipts=[ipt], service_account=service_account, network_policy=network_policy, + location_policy=location_policy, ) logs_policy = google_batch_operations.build_logs_policy( @@ -697,20 +789,22 @@ def _create_batch_request( # Bring together the task definition(s) and build the Job request. task_spec = google_batch_operations.build_task_spec( - runnables=runnables, volumes=[datadisk_volume] + runnables=runnables, volumes=([datadisk_volume] + gcs_volumes) ) task_group = google_batch_operations.build_task_group( - task_spec, all_envs, task_count=len(all_envs), task_count_per_node=1 + task_spec, task_count=1, task_count_per_node=1 ) job = google_batch_operations.build_job( [task_group], allocation_policy, labels, logs_policy ) + batch_job_id = self._format_batch_job_id(task_metadata, job_metadata) + job_request = batch_v1.CreateJobRequest( parent=f'projects/{self._project}/locations/{self._location}', job=job, - job_id=job_id, + job_id=batch_job_id, ) # pylint: enable=line-too-long return job_request @@ -722,43 +816,6 @@ def _submit_batch_job(self, request) -> str: print(f'Provider internal-id (operation): {job_response.name}') return op.get_field('task-id') - def _create_env_for_task( - self, task_view: job_model.JobDescriptor - ) -> Dict[str, str]: - job_params = task_view.job_params - task_params = task_view.task_descriptors[0].task_params - - # Set local variables for the core pipeline values - script = task_view.job_metadata['script'] - user_project = task_view.job_metadata['user-project'] or '' - - envs = job_params['envs'] | task_params['envs'] - inputs = job_params['inputs'] | task_params['inputs'] - outputs = job_params['outputs'] | task_params['outputs'] - mounts = job_params['mounts'] - - prepare_env = self._get_prepare_env( - script, task_view, inputs, outputs, mounts, _DATA_MOUNT_POINT - ) - localization_env = self._get_localization_env( - inputs, user_project, _DATA_MOUNT_POINT - ) - user_environment = self._build_user_environment( - envs, inputs, outputs, mounts, _DATA_MOUNT_POINT - ) - delocalization_env = self._get_delocalization_env( - outputs, user_project, _DATA_MOUNT_POINT - ) - # This merges all the envs into one dict. Need to use this syntax because - # of python3.6. In python3.9 we'd prefer to use | operator. - all_env = { - **prepare_env, - **localization_env, - **user_environment, - **delocalization_env, - } - return all_env - def submit_job( self, job_descriptor: job_model.JobDescriptor, @@ -776,22 +833,15 @@ def submit_job( # Prepare and submit jobs. launched_tasks = [] requests = [] - job_id = job_descriptor.job_metadata['job-id'] - # Instead of creating one job per task, create one job with several tasks. - # We also need to create a list of environments per task. The length of this - # list determines how many tasks are in the job, and is specified in the - # TaskGroup's task_count field. - envs = [] + for task_view in job_model.task_view_generator(job_descriptor): - env = self._create_env_for_task(task_view) - envs.append(google_batch_operations.build_environment(env)) + request = self._create_batch_request(task_view) + if self._dry_run: + requests.append(request) + else: + task_id = self._submit_batch_job(request) + launched_tasks.append(task_id) - request = self._create_batch_request(job_descriptor, job_id, envs) - if self._dry_run: - requests.append(request) - else: - task_id = self._submit_batch_job(request) - launched_tasks.append(task_id) # If this is a dry-run, emit all the batch request objects if self._dry_run: # Each request is a google.cloud.batch_v1.types.batch.CreateJobRequest @@ -800,7 +850,7 @@ def submit_job( # Ideally, we could serialize these request objects to yaml or json. print(requests) return { - 'job-id': job_id, + 'job-id': job_descriptor.job_metadata['job-id'], 'user-id': job_descriptor.job_metadata.get('user-id'), 'task-id': [task_id for task_id in launched_tasks if task_id], } @@ -884,8 +934,11 @@ def lookup_job_tasks( # Make the request response = client.list_jobs(request=request) - for page in response: - yield GoogleBatchOperation(page) + # Sort the operations by create-time to match sort of other providers + operations = [GoogleBatchOperation(page) for page in response] + operations.sort(key=lambda op: op.get_field('create-time'), reverse=True) + for op in operations: + yield op def get_tasks_completion_messages(self, tasks): # TODO: This needs to return a list of error messages for each task diff --git a/dsub/providers/google_batch_operations.py b/dsub/providers/google_batch_operations.py index 62042e5..5a26f9a 100644 --- a/dsub/providers/google_batch_operations.py +++ b/dsub/providers/google_batch_operations.py @@ -33,13 +33,14 @@ def get_label(op: batch_v1.types.Job, name: str) -> str: return op.labels.get(name) -def get_environment(op: batch_v1.types.Job) -> Dict[str, str]: +def get_environment( + op: batch_v1.types.Job, runnable_index: int +) -> Dict[str, str]: # Currently Batch only supports task_groups of size 1 task_group = op.task_groups[0] - env_dict = {} - for env in task_group.task_environments: - env_dict.update(env.variables) - return env_dict + task_spec = task_group.task_spec + runnables = task_spec.runnables + return runnables[runnable_index].environment.variables def is_done(op: batch_v1.types.Job) -> bool: @@ -139,7 +140,6 @@ def build_environment(env_vars: Dict[str, str]): def build_task_group( task_spec: batch_v1.types.TaskSpec, - task_environments: List[batch_v1.types.Environment], task_count: int, task_count_per_node: int, ) -> batch_v1.types.TaskGroup: @@ -147,7 +147,6 @@ def build_task_group( Args: task_spec (TaskSpec): TaskSpec object - task_environments (List[Environment]): List of Environment objects task_count (int): The number of total tasks in the job task_count_per_node (int): The number of tasks to schedule on one VM @@ -156,7 +155,6 @@ def build_task_group( """ task_group = batch_v1.TaskGroup() task_group.task_spec = task_spec - task_group.task_environments = task_environments task_group.task_count = task_count task_group.task_count_per_node = task_count_per_node return task_group @@ -222,6 +220,26 @@ def build_volume(disk: str, path: str) -> batch_v1.types.Volume: return volume +def build_gcs_volume( + bucket: str, path: str, mount_options: List[str] +) -> batch_v1.types.Volume: + """Build a Volume object mounted to a GCS bucket for a Batch request. + + Args: + bucket (str): Name of bucket to mount (without the gs:// prefix) + path (str): Path to mount the bucket at inside the container. + mount_options (List[str]): List of mount options + + Returns: + An object representing a Mount. + """ + volume = batch_v1.Volume() + volume.gcs = batch_v1.GCS(remote_path=bucket) + volume.mount_path = path + volume.mount_options = mount_options + return volume + + def build_network_policy( network: str, subnetwork: str, @@ -264,12 +282,13 @@ def build_allocation_policy( ipts: List[batch_v1.types.AllocationPolicy.InstancePolicyOrTemplate], service_account: batch_v1.types.ServiceAccount, network_policy: batch_v1.types.AllocationPolicy.NetworkPolicy, + location_policy: batch_v1.types.AllocationPolicy.LocationPolicy, ) -> batch_v1.types.AllocationPolicy: allocation_policy = batch_v1.AllocationPolicy() allocation_policy.instances = ipts allocation_policy.service_account = service_account allocation_policy.network = network_policy - + allocation_policy.location = location_policy return allocation_policy @@ -348,3 +367,11 @@ def build_accelerators( accelerators.append(accelerator) return accelerators + + +def build_location_policy( + allowed_locations: List[str], +) -> batch_v1.types.AllocationPolicy.LocationPolicy: + location_policy = batch_v1.AllocationPolicy.LocationPolicy() + location_policy.allowed_locations = allowed_locations + return location_policy diff --git a/test/integration/test_setup.sh b/test/integration/test_setup.sh index 9cad298..b8b1763 100644 --- a/test/integration/test_setup.sh +++ b/test/integration/test_setup.sh @@ -34,7 +34,8 @@ # If the script name is ..sh, pull out the provider. # If the script name is .sh, use "local". # If the DSUB_PROVIDER is set, make sure it is correct for a provider test. -# Special-case the google-v2 tests to be runnable for google-cls-v2. +# Special-case the google-v2 tests to be runnable for google-cls-v2 +# and google-batch. readonly SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_DEFAULT_PROVIDER=$( @@ -48,6 +49,9 @@ elif [[ -n "${SCRIPT_DEFAULT_PROVIDER}" ]]; then if [[ "${DSUB_PROVIDER}" == "google-cls-v2" ]] && \ [[ "${SCRIPT_DEFAULT_PROVIDER}" == "google-v2" ]]; then echo "Running google-v2 e2e/unit tests with provider google-cls-v2" + elif [[ "${DSUB_PROVIDER}" == "google-batch" ]] && \ + [[ "${SCRIPT_DEFAULT_PROVIDER}" == "google-v2" ]]; then + echo "Running google-v2 e2e/unit tests with provider google-batch" elif [[ "${DSUB_PROVIDER}" != "${SCRIPT_DEFAULT_PROVIDER}" ]]; then 1>&2 echo "DSUB_PROVIDER is '${DSUB_PROVIDER:-}' not '${SCRIPT_DEFAULT_PROVIDER}'" exit 1 diff --git a/test/integration/unit_flags.google-batch.sh b/test/integration/unit_flags.google-batch.sh index 1433ecd..ac8724e 100755 --- a/test/integration/unit_flags.google-batch.sh +++ b/test/integration/unit_flags.google-batch.sh @@ -258,7 +258,124 @@ function test_network() { } readonly -f test_network -# Run the tests +function test_location() { + local subtest="${FUNCNAME[0]}" + + if call_dsub \ + --location us-west2 \ + --command 'echo "${TEST_NAME}"'; then + + # Check that the output contains expected values + location_result=$(grep " allowed_locations:" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${location_result}" != "regions/us-west2" ]]; then + 1>&2 echo "location was actually ${location_result}, expected regions/us-west2" + exit 1 + fi + + test_passed "${subtest}" + else + 1>&2 echo "Using the location flag generated an error" + + test_failed "${subtest}" + fi +} +readonly -f test_location + +function test_neither_region_nor_zone() { + local subtest="${FUNCNAME[0]}" + + if call_dsub \ + --command 'echo "${TEST_NAME}"'; then + + # Check that the output contains expected values + location_result=$(grep " allowed_locations:" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${location_result}" != "regions/us-central1" ]]; then + 1>&2 echo "location was actually ${location_result}, expected regions/us-central1" + exit 1 + fi + + test_passed "${subtest}" + else + 1>&2 echo "Location not used as default region" + + test_failed "${subtest}" + fi +} +readonly -f test_neither_region_nor_zone + +function test_region_and_zone() { + local subtest="${FUNCNAME[0]}" + + if call_dsub \ + --command 'echo "${TEST_NAME}"' \ + --zones us-central1-f \ + --regions us-central1; then + + # Check that the output contains expected values + regions_result=$(grep " allowed_locations: \"regions/" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${regions_result}" != "regions/us-central1" ]]; then + 1>&2 echo "location was actually ${regions_result}, expected regions/us-central1" + exit 1 + fi + + zones_result=$(grep " allowed_locations: \"zones/" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${zones_result}" != "zones/us-central1-f" ]]; then + 1>&2 echo "location was actually ${zones_result}, expected zones/us-central1-f" + exit 1 + fi + + test_passed "${subtest}" + else + 1>&2 echo "Location not used as default region" + + test_failed "${subtest}" + fi +} +readonly -f test_neither_region_nor_zone + +function test_regions() { + local subtest="${FUNCNAME[0]}" + + if call_dsub \ + --command 'echo "${TEST_NAME}"' \ + --regions us-central1; then + + # Check that the output contains expected values + location_result=$(grep " allowed_locations:" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${location_result}" != "regions/us-central1" ]]; then + 1>&2 echo "location was actually ${location_result}, expected regions/us-central1" + exit 1 + fi + + test_passed "${subtest}" + else + test_failed "${subtest}" + fi +} +readonly -f test_regions + +function test_zones() { + local subtest="${FUNCNAME[0]}" + + if call_dsub \ + --command 'echo "${TEST_NAME}"' \ + --zones us-central1-a; then + + # Check that the output contains expected values + location_result=$(grep " allowed_locations:" "${TEST_STDERR}" | awk -F\" '{print $2}') + if [[ "${location_result}" != "zones/us-central1-a" ]]; then + 1>&2 echo "location was actually ${location_result}, expected zones/us-central1-a" + exit 1 + fi + + test_passed "${subtest}" + else + test_failed "${subtest}" + fi +} +readonly -f test_zones + +# # Run the tests trap "exit_handler" EXIT mkdir -p "${TEST_TMP}" @@ -284,3 +401,10 @@ test_no_service_account echo test_network + +echo +test_location +test_neither_region_nor_zone +test_region_and_zone +test_regions +test_zones diff --git a/test/run_tests.sh b/test/run_tests.sh index f3d5dc6..f9683ae 100755 --- a/test/run_tests.sh +++ b/test/run_tests.sh @@ -226,10 +226,15 @@ function get_test_providers() { e2e_command_flag.sh | \ e2e_dsub_summary.sh | \ e2e_env_list.py | \ + e2e_env_tasks.sh | \ e2e_image.sh | \ e2e_input_wildcards.sh | \ e2e_io.sh | \ + e2e_io_auto.sh | \ + e2e_io_gcs_tasks.sh | \ + e2e_io_mount_bucket.google-v2.sh | \ e2e_io_recursive.sh | \ + e2e_io_tasks.sh | \ e2e_logging_content.sh | \ e2e_logging_fail.sh | \ e2e_logging_paths.sh | \ @@ -239,6 +244,9 @@ function get_test_providers() { e2e_non_root.sh | \ e2e_python.sh | \ e2e_requester_pays_buckets.sh | \ + e2e_retries_success.sh | \ + e2e_retries_fail_1.sh | \ + e2e_retries_fail_2.sh | \ e2e_runtime.sh) local all_provider_list="${DSUB_PROVIDER:-local google-v2 google-cls-v2 google-batch}" ;;