feat: Add SAM Pipeline commands (aws#3085)

* sam pipeline bootstrap (aws#2811) * two-stages-pipeline plugin * typos * add docstring * make mypy happy * removing swap file * delete the two_stages_pipeline plugin as the pipeline-bootstrap command took over its responsibility * remove 'get_template_function_runtimes' function as the decision is made to not process the SAM template during pipeline init which was the only place we use the function * sam pipeline bootstrap command * move the pipelineconfig.toml file to .aws-sam * UX - rewriting Co-authored-by: Chris Rehn <crehn@outlook.com> * UX improvements * make black happy * apply review comments * UX - rewriting Co-authored-by: Chris Rehn <crehn@outlook.com> * refactor * Apply review comments * use python way of array elements assignments * Update samcli/lib/pipeline/bootstrap/stage.py Co-authored-by: _sam <3804518+aahung@users.noreply.github.com> * apply review comments * typo * read using utf-8 * create and user a safe version of the save_config method * apply review comments * rename _get_command_name to _get_command_names * don't save generated ARNs for now, will save during init * Revert "don't save generated ARNs for now, will save during init" This reverts commit d184e164022d9560131c62a826436edbc93da189. * Notify the user to rotate periodically rotate the IAM credentials * typo * Use AES instead of KMS for S3 SSE * rename Ecr to ECR and Iam to IAM * Grant lambda service explicit permissions to thhe ECR instead of relying on giving this permissions on ad-hoc while creating the container images Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: _sam <3804518+aahung@users.noreply.github.com> * sam pipeline init command (aws#2831) * sam pipeline init command * apply review comments * apply review comments * display a message that we have successfully created the pipeline configuration file(s). * doc typo * Let 'sam pipeline init' prefills pipeline's infrastructure resources… (aws#2894) * Let 'sam pipeline init' prefills pipeline's infrastructure resources' values from 'sam pipeline bootstrap' results. * save bootstrapped sateg region * make black happy * exclude non-dict keys from samconfig.get_env_names method. * Rename the pipeline 'Stage' concept to 'Environment' (aws#2908) * Rename the pipeline 'Stage' concept to 'Environment' * typo * Rename --environment-name argument to --environment * Sam pipelines ux rename ecr repo to image repository (aws#2910) * Rename ecr-repo to image-repository * UT Fixes * typo * typo * feat: Support creating pipeline files directly into . without hooks (aws#2911) * feat: Support creating pipeline files directly into . without hooks * Integration test for pipeline init and pipeline bootstrap (aws#2841) * Expose Environment._get_stack_name for integ test to predict stack name * Add integ test for pipeline bootstrap * Add init integ test * small UX improvements: (aws#2914) * small UX improvements: 1. show a message when the user cancels a bootstrapping command. 2. Don't prompt for CI/CD provider or provider templates if there is only one choice. 3. Make PipelineFileAlreadyExistsError a UserError. 4. use the Colored class instead of fg='color' when prompting a colored message. 5. Fix a bug where we were not allowing empty response for not required questions. * Fix Integration Test: We now don't ask the user to select a provider's pipeline template if there is only one * Add docs for PipelineFileAlreadyExistsError * make black happy * Sam pipelines s3 security (aws#2975) * Deny non https requests for the artifacts S3 bucket * enable bucket serverside logging * add integration tests for artifacts bucket SSL-only requests and access logging * typo * Ensure the ArtifactsLoggingBucket denies non ssl requests (aws#2976) * Sam pipelines ux round 3 (aws#2979) * rename customer facing message 'CI/CD provider' to 'CI/CD system' * add a note about what 'Environment Name' is during the pipeline bootstrap guided context * Apply suggestions from code review typo Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * let pipeline IAM user assume only IAM roles tagged with Role=pipeline-execution-role (aws#2982) * Adding AWS_ prefix to displayed out. (aws#2993) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Add region to pipeline bootstrap interactive flow (aws#2997) * Ask AWS region in bootstrap interactive flow * Read default region from boto session first * Fix a unit test * Inform write to pipelineconfig.toml at the end of bootstrap (aws#3002) * Print info about pipelineconfig.toml after resources are bootstrapped * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * List detected env names in pipeline init when prompt to input the env name (aws#3000) * Allow question.question can be resolved using key path * Pass the list of env names message (environment_names_message) into pipeline init interactive flow context * Update samcli/commands/pipeline/init/interactive_init_flow.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Fix unit test (trigger pr builds) * Fix integ test * Fix integ test Co-authored-by: Chris Rehn <crehn@outlook.com> * Adding account id to bootstrap message. (aws#2998) * Adding account id to bootstrap message. * adding docstring * Addressing PR comments. * Adding unit tests. * Fixing unit tests. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Cfn creds fix (aws#3014) * Removing pipeline user creds from cfn output. This maintains same user exp. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Ux bootstrap revamp 20210706 (aws#3021) * Add intro paragraph to bootstrap * Add switch account prompt * Revamp stage definition prompt * Revamp existing resources prompt * Revamp security prompt * Allow answers to be changed later * Add exit message for bootstrap * Add exit message for bootstrap (1) * Add indentation to review values * Add "Below is the summary of the answers:" * Sweep pylint errors * Update unit tests * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/guided_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update samcli/commands/pipeline/bootstrap/cli.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Update unit tests * Add bold to other literals Co-authored-by: Chris Rehn <crehn@outlook.com> * Adding account condition for CFN execution role. (aws#3027) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * pipeline UX revamp 20210707 (aws#3031) * Allow running bootstrap inside pipeline init * Select account credential source within bootstrap * Add bootstrap decorations within pipeline init * Removing ip range option from bootstrap. (aws#3036) * Removing ip range option from bootstrap. * Fixing unit test from UX PR. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix toml file incorrect read/write in init --bootstrap (aws#3037) * Temporarily removing account fix. (aws#3038) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Rename environment to stage (aws#3040) * Improve account source selection (aws#3042) * Fixing various cosmetics UX issues with pipeline workflow. (aws#3046) * Fixing credential to credentials * Forcing text color to yellow. * Adding new line after stage diagram. * Adding extra line after checking bootstrap message. * Renaming config -> configuration * account source -> credential source * Removing old message. * Fixing indentation in list. * Fixing bunch of indentation. * fixing f string Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Auto skip questions if stage detected (aws#3045) * Autofill question if default value is presented * Allow to use index to select stage names (aws#3051) * Updating message when bootstrap stages are missing. (aws#3058) * Updating message when bootstrap stages are missing. * Fixing indendation Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fixing bootstrap integ tests. (aws#3061) * Fixing bootstrap integ tests. * Cleaning up some integ tests. * Using environment variables when running integ test on CI. * Using expression instead of full loop. * Adding instruction to use default profile on local. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix bootstrap test region (#3064) * Fix bootstrap region in integ test * Fix regions in non-interactive mode as well * Add more pipeline init integ test (aws#3065) * Fix existing pipeline init integ test * Add more pipeline init integ tests * Config file bug (aws#3066) * Validating config file after bootstrap stack creation. * Validating config file after bootstrap. Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> * Fix pipeline init integ test because of pipelineconfig file exists (aws#3067) * Make stage name randomized to avoid race condition among multi canary runs (aws#3078) * Load number of stages from pipeline template (aws#3059) * Load number of stages from templates * Rename variable and add debug log * Add encoding to open() * Allow roles with Tag aws-sam-pipeline-codebuild-service-role to assume PipelineExecutionRole (aws#2950) * pipeline init UX: Ask to confirm when file exists (aws#3079) * Ask to confirm overriding if files already exist, or save to another directory * Add doc links (aws#3087) * Adding accidentally removed tests back. (aws#3088) Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com> Co-authored-by: elbayaaa <72949274+elbayaaa@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Ahmed Elbayaa <elbayaaa@amazon.com> Co-authored-by: Tarun <c2tarun@users.noreply.github.com> Co-authored-by: Tarun Mall <tarun@amazon.noreply.github.com>
thoward-godaddy · Jul 20, 2021 · 0626f94 · 0626f94
1 parent 59c8576
commit 0626f94
Show file tree

Hide file tree

Showing 64 changed files with 5,052 additions and 113 deletions.
diff --git a/mypy.ini b/mypy.ini
@@ -59,6 +59,6 @@ ignore_missing_imports=True
 ignore_missing_imports=True
 
 # progressive add typechecks and these modules already complete the process, let's keep them clean
-[mypy-samcli.commands.build,samcli.lib.build.*,samcli.commands.local.cli_common.invoke_context,samcli.commands.local.lib.local_lambda,samcli.lib.providers.*,samcli.lib.utils.git_repo.py]
+[mypy-samcli.commands.build,samcli.lib.build.*,samcli.commands.local.cli_common.invoke_context,samcli.commands.local.lib.local_lambda,samcli.lib.providers.*,samcli.lib.utils.git_repo.py,samcli.lib.cookiecutter.*,samcli.lib.pipeline.*,samcli.commands.pipeline.*]
 disallow_untyped_defs=True
 disallow_incomplete_defs=True
diff --git a/samcli/cli/command.py b/samcli/cli/command.py
@@ -21,6 +21,7 @@
     "samcli.commands.deploy",
     "samcli.commands.logs",
     "samcli.commands.publish",
+    "samcli.commands.pipeline.pipeline",
     # We intentionally do not expose the `bootstrap` command for now. We might open it up later
     # "samcli.commands.bootstrap",
 ]

diff --git a/samcli/cli/context.py b/samcli/cli/context.py
@@ -4,7 +4,7 @@
 
 import logging
 import uuid
-from typing import Optional, cast
+from typing import Optional, cast, List
 
 import boto3
 import botocore
@@ -186,7 +186,7 @@ def _refresh_session(self):
             raise CredentialsError(str(ex)) from ex
 
 
-def get_cmd_names(cmd_name, ctx):
+def get_cmd_names(cmd_name, ctx) -> List[str]:
     """
     Given the click core context, return a list representing all the subcommands passed to the CLI
 

diff --git a/samcli/commands/_utils/template.py b/samcli/commands/_utils/template.py
@@ -9,16 +9,16 @@
 import yaml
 from botocore.utils import set_value_from_jmespath
 
-from samcli.commands.exceptions import UserException
-from samcli.lib.utils.packagetype import ZIP
-from samcli.yamlhelper import yaml_parse, yaml_dump
 from samcli.commands._utils.resources import (
     METADATA_WITH_LOCAL_PATHS,
     RESOURCES_WITH_LOCAL_PATHS,
     AWS_SERVERLESS_FUNCTION,
     AWS_LAMBDA_FUNCTION,
     get_packageable_resource_paths,
 )
+from samcli.commands.exceptions import UserException
+from samcli.lib.utils.packagetype import ZIP
+from samcli.yamlhelper import yaml_parse, yaml_dump
 
 
 class TemplateNotFoundException(UserException):

diff --git a/samcli/commands/deploy/guided_context.py b/samcli/commands/deploy/guided_context.py
@@ -6,7 +6,6 @@
 from typing import Dict, Any, List
 
 import click
-from botocore.session import get_session
 from click import confirm
 from click import prompt
 from click.types import FuncParamType
@@ -36,6 +35,7 @@
 from samcli.lib.providers.sam_function_provider import SamFunctionProvider
 from samcli.lib.providers.sam_stack_provider import SamLocalStackProvider
 from samcli.lib.utils.colors import Colored
+from samcli.lib.utils.defaults import get_default_aws_region
 from samcli.lib.utils.packagetype import IMAGE
 
 LOG = logging.getLogger(__name__)
@@ -110,7 +110,7 @@ def guided_prompts(self, parameter_override_keys):
             The keys of parameters to override, for each key, customers will be asked to provide a value
         """
         default_stack_name = self.stack_name or "sam-app"
-        default_region = self.region or get_session().get_config_variable("region") or "us-east-1"
+        default_region = self.region or get_default_aws_region()
         default_capabilities = self.capabilities[0] or ("CAPABILITY_IAM",)
         default_config_env = self.config_env or DEFAULT_ENV
         default_config_file = self.config_file or DEFAULT_CONFIG_FILE_NAME

diff --git a/samcli/commands/exceptions.py b/samcli/commands/exceptions.py
@@ -59,3 +59,22 @@ class ContainersInitializationException(UserException):
     """
     Exception class when SAM is not able to initialize any of the lambda functions containers
     """
+
+
+class PipelineTemplateCloneException(UserException):
+    """
+    Exception class when unable to download pipeline templates from a Git repository during `sam pipeline init`
+    """
+
+
+class AppPipelineTemplateManifestException(UserException):
+    """
+    Exception class when SAM is not able to parse the "manifest.yaml" file located in the SAM pipeline templates
+    Git repo: "github.com/aws/aws-sam-cli-pipeline-init-templates.git
+    """
+
+
+class AppPipelineTemplateMetadataException(UserException):
+    """
+    Exception class when SAM is not able to parse the "metadata.json" file located in the SAM pipeline templates
+    """
diff --git a/samcli/commands/pipeline/__init__.py b/samcli/commands/pipeline/__init__.py
diff --git a/samcli/commands/pipeline/bootstrap/__init__.py b/samcli/commands/pipeline/bootstrap/__init__.py
diff --git a/samcli/commands/pipeline/bootstrap/cli.py b/samcli/commands/pipeline/bootstrap/cli.py
@@ -0,0 +1,238 @@
+"""
+CLI command for "pipeline bootstrap", which sets up the require pipeline infrastructure resources
+"""
+import os
+from textwrap import dedent
+from typing import Any, Dict, List, Optional
+
+import click
+
+from samcli.cli.cli_config_file import configuration_option, TomlProvider
+from samcli.cli.main import pass_context, common_options, aws_creds_options, print_cmdline_args
+from samcli.lib.config.samconfig import SamConfig
+from samcli.lib.pipeline.bootstrap.stage import Stage
+from samcli.lib.telemetry.metric import track_command
+from samcli.lib.utils.colors import Colored
+from samcli.lib.utils.version_checker import check_newer_version
+from .guided_context import GuidedContext
+from ..external_links import CONFIG_AWS_CRED_ON_CICD_URL
+
+SHORT_HELP = "Generates the necessary AWS resources to connect your CI/CD system."
+
+HELP_TEXT = """
+SAM Pipeline Bootstrap generates the necessary AWS resources to connect your
+CI/CD system. This step must be completed for each pipeline stage prior to
+running sam pipeline init
+"""
+
+PIPELINE_CONFIG_DIR = os.path.join(".aws-sam", "pipeline")
+PIPELINE_CONFIG_FILENAME = "pipelineconfig.toml"
+
+
+@click.command("bootstrap", short_help=SHORT_HELP, help=HELP_TEXT, context_settings=dict(max_content_width=120))
+@configuration_option(provider=TomlProvider(section="parameters"))
+@click.option(
+    "--interactive/--no-interactive",
+    is_flag=True,
+    default=True,
+    help="Disable interactive prompting for bootstrap parameters, and fail if any required arguments are missing.",
+)
+@click.option(
+    "--stage",
+    help="The name of the corresponding stage. It is used as a suffix for the created resources.",
+    required=False,
+)
+@click.option(
+    "--pipeline-user",
+    help="An IAM user generated or referenced by sam pipeline bootstrap in order to "
+    "allow the connected CI/CD system to connect to the SAM CLI.",
+    required=False,
+)
+@click.option(
+    "--pipeline-execution-role",
+    help="Execution role that the CI/CD system assumes in order to make changes to resources on your behalf.",
+    required=False,
+)
+@click.option(
+    "--cloudformation-execution-role",
+    help="Execution role that CloudFormation assumes in order to make changes to resources on your behalf",
+    required=False,
+)
+@click.option(
+    "--bucket",
+    help="The name of the S3 bucket where this command uploads your CloudFormation template. This is required for"
+    "deployments of templates sized greater than 51,200 bytes.",
+    required=False,
+)
+@click.option(
+    "--create-image-repository/--no-create-image-repository",
+    is_flag=True,
+    default=False,
+    help="If set to true and no ECR image repository is provided, this command will create an ECR image repository "
+    "to hold the container images of Lambda functions having an Image package type.",
+)
+@click.option(
+    "--image-repository",
+    help="ECR repo uri where this command uploads the image artifacts that are referenced in your template.",
+    required=False,
+)
+@click.option(
+    "--confirm-changeset/--no-confirm-changeset",
+    default=True,
+    is_flag=True,
+    help="Prompt to confirm if the resources is to be deployed by SAM CLI.",
+)
+@common_options
+@aws_creds_options
+@pass_context
+@track_command
+@check_newer_version
+@print_cmdline_args
+def cli(
+    ctx: Any,
+    interactive: bool,
+    stage: Optional[str],
+    pipeline_user: Optional[str],
+    pipeline_execution_role: Optional[str],
+    cloudformation_execution_role: Optional[str],
+    bucket: Optional[str],
+    create_image_repository: bool,
+    image_repository: Optional[str],
+    confirm_changeset: bool,
+    config_file: Optional[str],
+    config_env: Optional[str],
+) -> None:
+    """
+    `sam pipeline bootstrap` command entry point
+    """
+    do_cli(
+        region=ctx.region,
+        profile=ctx.profile,
+        interactive=interactive,
+        stage_name=stage,
+        pipeline_user_arn=pipeline_user,
+        pipeline_execution_role_arn=pipeline_execution_role,
+        cloudformation_execution_role_arn=cloudformation_execution_role,
+        artifacts_bucket_arn=bucket,
+        create_image_repository=create_image_repository,
+        image_repository_arn=image_repository,
+        confirm_changeset=confirm_changeset,
+        config_file=config_env,
+        config_env=config_file,
+    )  # pragma: no cover
+
+
+def do_cli(
+    region: Optional[str],
+    profile: Optional[str],
+    interactive: bool,
+    stage_name: Optional[str],
+    pipeline_user_arn: Optional[str],
+    pipeline_execution_role_arn: Optional[str],
+    cloudformation_execution_role_arn: Optional[str],
+    artifacts_bucket_arn: Optional[str],
+    create_image_repository: bool,
+    image_repository_arn: Optional[str],
+    confirm_changeset: bool,
+    config_file: Optional[str],
+    config_env: Optional[str],
+    standalone: bool = True,
+) -> None:
+    """
+    implementation of `sam pipeline bootstrap` command
+    """
+    if not pipeline_user_arn:
+        pipeline_user_arn = _load_saved_pipeline_user_arn()
+
+    if interactive:
+        if standalone:
+            click.echo(
+                dedent(
+                    """\
+
+                    sam pipeline bootstrap generates the necessary AWS resources to connect a stage in
+                    your CI/CD system. We will ask for [1] stage definition, [2] account details, and
+                    [3] references to existing resources in order to bootstrap these pipeline
+                    resources.
+                    """
+                ),
+            )
+
+        guided_context = GuidedContext(
+            profile=profile,
+            stage_name=stage_name,
+            pipeline_user_arn=pipeline_user_arn,
+            pipeline_execution_role_arn=pipeline_execution_role_arn,
+            cloudformation_execution_role_arn=cloudformation_execution_role_arn,
+            artifacts_bucket_arn=artifacts_bucket_arn,
+            create_image_repository=create_image_repository,
+            image_repository_arn=image_repository_arn,
+            region=region,
+        )
+        guided_context.run()
+        stage_name = guided_context.stage_name
+        pipeline_user_arn = guided_context.pipeline_user_arn
+        pipeline_execution_role_arn = guided_context.pipeline_execution_role_arn
+        cloudformation_execution_role_arn = guided_context.cloudformation_execution_role_arn
+        artifacts_bucket_arn = guided_context.artifacts_bucket_arn
+        create_image_repository = guided_context.create_image_repository
+        image_repository_arn = guided_context.image_repository_arn
+        region = guided_context.region
+        profile = guided_context.profile
+
+    if not stage_name:
+        raise click.UsageError("Missing required parameter '--stage'")
+
+    environment: Stage = Stage(
+        name=stage_name,
+        aws_profile=profile,
+        aws_region=region,
+        pipeline_user_arn=pipeline_user_arn,
+        pipeline_execution_role_arn=pipeline_execution_role_arn,
+        cloudformation_execution_role_arn=cloudformation_execution_role_arn,
+        artifacts_bucket_arn=artifacts_bucket_arn,
+        create_image_repository=create_image_repository,
+        image_repository_arn=image_repository_arn,
+    )
+
+    bootstrapped: bool = environment.bootstrap(confirm_changeset=confirm_changeset)
+
+    if bootstrapped:
+        environment.print_resources_summary()
+
+        environment.save_config_safe(
+            config_dir=PIPELINE_CONFIG_DIR, filename=PIPELINE_CONFIG_FILENAME, cmd_names=_get_bootstrap_command_names()
+        )
+
+        click.secho(
+            dedent(
+                f"""\
+                View the definition in {os.path.join(PIPELINE_CONFIG_DIR, PIPELINE_CONFIG_FILENAME)},
+                run sam pipeline bootstrap to generate another set of resources, or proceed to
+                sam pipeline init to create your pipeline configuration file.
+                """
+            )
+        )
+
+        if not environment.pipeline_user.is_user_provided:
+            click.secho(
+                dedent(
+                    f"""\
+                    Before running {Colored().bold("sam pipeline init")}, we recommend first setting up AWS credentials
+                    in your CI/CD account. Read more about how to do so with your provider in
+                    {CONFIG_AWS_CRED_ON_CICD_URL}.
+                    """
+                )
+            )
+
+
+def _load_saved_pipeline_user_arn() -> Optional[str]:
+    samconfig: SamConfig = SamConfig(config_dir=PIPELINE_CONFIG_DIR, filename=PIPELINE_CONFIG_FILENAME)
+    if not samconfig.exists():
+        return None
+    config: Dict[str, str] = samconfig.get_all(cmd_names=_get_bootstrap_command_names(), section="parameters")
+    return config.get("pipeline_user")
+
+
+def _get_bootstrap_command_names() -> List[str]:
+    return ["pipeline", "bootstrap"]