Docs for 0.2.0

HISTORY.rst

@@ -2,6 +2,40 @@
 History
 =======
 
+0.2.0 (2020-05-01)
+==================
+
+Code refactored to allow a better way to go over each template and stack so everything is treated in memory
+before being put into a file and uploaded into S3.
+
+* Issues closed
+    * Docs update and first go at IAM perms (`#22`_)
+    * Refactor of XModules logic onto ECS services (`#39`_)
+    * Templates & Stacks refactor (`#38`_)
+    * Update issue templates for easy PRs and Bug reports
+    * Added `make conform` to run black against the code to standardize syntax (`#26`_)
+    * Allow to specify directory to write all the templates to in addition to S3. (`#27`_)
+    * Reformatted with black (`#25`_)
+    * Expand TagsSpecifications with x-tags (`#24`_)
+    * Bug fix for root template and Cluster reference (`#20`_)
+
+Documentation structure and content updated to help navigate through modules in an easier way.
+Documented syntax reference for each module
+
+New feature
+^^^^^^^^^^^
+
+`#6`_ - Implement x-rds. Allows to create RDS databases with very little properties needed
+
+* Creates Aurora cluster and DB Instance
+* Creates the DB Parameter Group by importing default settings.
+* Creates a common subnet group for all DBs to run into (goes to Storage subnets when using --create-vpc).
+* Creates DB username and password in AWS SecretsManager
+* Applies IAM permissions to ECS Execution Role to get access to the secret
+* Applies ECS Container Secrets to the containers to provide them with the secret values through Environment variables.
+
+
+
 0.1.3 (2020-04-13)
 ==================
 
@@ -10,6 +44,9 @@ a CICD pipeline.
 
 See overall progress on `GH Project`_
 
+Issues closed
+^^^^^^^^^^^^^
+
 * `Issue 14 <https://github.com/lambda-my-aws/ecs_composex/issues/14>`_
 * `Issue 15 <https://github.com/lambda-my-aws/ecs_composex/issues/15>`_
 
@@ -20,6 +57,9 @@ See overall progress on `GH Project`_
 Patch release aiming to improve the CLI and integration of the Compute layer so that the compute resources creation
 in EC2 are standalone and can be created separately if one so wished to reuse.
 
+Issues closed
+^^^^^^^^^^^^^
+
  `Issue <https://github.com/lambda-my-aws/ecs_composex/issues/7>`_ related to the fix.
 
  `PR <https://github.com/lambda-my-aws/ecs_composex/pull/8>`_ related to the fix.
@@ -62,3 +102,13 @@ or alternatively in an object/dict format
 
 
 .. _GH Project: https://github.com/orgs/lambda-my-aws/projects/3
+
+.. _#22: https://github.com/lambda-my-aws/ecs_composex/issues/22
+.. _#39: https://github.com/lambda-my-aws/ecs_composex/issues/39
+.. _#38: https://github.com/lambda-my-aws/ecs_composex/issues/38
+.. _#27: https://github.com/lambda-my-aws/ecs_composex/issues/27
+.. _#26: https://github.com/lambda-my-aws/ecs_composex/issues/26
+.. _#25: https://github.com/lambda-my-aws/ecs_composex/issues/25
+.. _#24: https://github.com/lambda-my-aws/ecs_composex/issues/24
+.. _#20: https://github.com/lambda-my-aws/ecs_composex/issues/20
+.. _#6: https://github.com/lambda-my-aws/ecs_composex/issues/6

README.rst

@@ -145,7 +145,13 @@ IAM Permissions to execute ECS ComposeX
             Effect: Allow
             Action:
               - 's3:GetObject*'
-              - 's3:PutObject*'
+              - 's3:PutObject*
+          - Sid: Ec2Access
+            Effect: Allow
+            Action:
+              - ec2:DescribeAvailabilityZones
+            Resource:
+              - "*"
 
 
 Why though?
@@ -176,9 +182,9 @@ Why am I not using AWS CDK?
 
 I started this work before AWS CDK came out with any python support, and I am not a developer professionally but I do love developing, and python is my language
 of choice. Troposphere was the obvious choice as the python library to use to build all the CFN templates. I find the way Troposphere has been built is awesome,
-it has a very nice community and is released often. I did a few PRs myself and `Mark Peek`_ is very proactive with PRs, releases come out often.
-
-Will I use CDK in the future? Depends on how many of you are going to use ECS ComposeX and will ask for it.
+the name of the properties are the same as they are in AWS CloudFormation, which gives a sense of standard to the user,
+allowing an experience as close to copy-paste as possible. `Troposphere`_ has a very nice community and is released often.
+I did a few PRs myself and `Mark Peek`_ is very proactive with PRs, releases come out often.
 
 
 Why not stick to AWS CFN Templates and CFN macros ?
@@ -227,7 +233,7 @@ Then, with the newly released CFN Private Registries, mutate this system to have
 License and documentation
 ==========================
 
-* Free software: BSD license
+* Free software: GPLv3+
 * Documentation:
     * https://docs.ecs-composex.lambda-my-aws.io
     * https://ecs-composex.readthedocs.io/en/latest
@@ -239,7 +245,6 @@ This package would not have been possible without the amazing job done by the AW
 
 This package would not have been possible without the amazing community around `Troposphere`_!
 
-
 This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
 
 Disclaimer

docs/index.rst

@@ -2,9 +2,8 @@ Welcome to ECS-ComposeX's documentation!
 ========================================
 
 .. toctree::
-
     :maxdepth: 2
-    :caption: Contents:
+    :caption: Contents
 
     readme
     installation
@@ -26,7 +25,7 @@ Welcome to ECS-ComposeX's documentation!
 
 .. toctree::
     :maxdepth: 2
-    :caption: contributions
+    :caption: Contributions
 
     authors
 

docs/modules_syntax.rst

@@ -2,4 +2,12 @@
 ComposeX Modules Syntax
 =======================
 
+.. contents::
+
+.. include:: ../ecs_composex/COMMON_SYNTAX.rst
+
+.. include:: ../ecs_composex/sqs/SYNTAX.rst
+
 .. include:: ../ecs_composex/rds/SYNTAX.rst
+
+.. include:: ../ecs_composex/compute/SYNTAX.rst

ecs_composex/COMMON_SYNTAX.rst

@@ -0,0 +1,66 @@
+ecs_composex
+=============
+
+ECS ComposeX requires to expands onto the original Docker compose file defintion in order to map the docker compose
+properties to their equivalent settings on AWS ECS and otherwise for the other "Extra" resources.
+
+In general for each x- section of the docker compose document, we will find three attributes to each resource:
+
+* Settings
+* Services
+* Properties
+
+Settings
+--------
+
+The settings is the section where we can take shortcuts or wrap around settings which would otherwise be complex to
+define. Sometimes, it simply is an easy way to use helpers which are configurable. For example, in the next interation
+for the x-rds resources, we will allow to define the latest RDS engine and version that supports Serverless for aurora.
+
+There is a set of settings which are going to be generic to all modules.
+
+EnvNames
+^^^^^^^^^
+Multiple teams who would want to adopt ECS ComposeX might already have their own environment variable keys (or names)
+for a common resource. For example, team A and team B can use the same SQS queue but they did not define a common name
+for it, so team A calls it *QueueA* and team B calls it *QUEUE_A*.
+
+With EnvNames, you can define a list of environment variables that will all share the same value, simply have a different
+name.
+
+.. hint::
+
+    No need to add the name of the resource as defined in the docker compose file, this will always be added by default.
+
+Services
+--------
+
+This is a list of object, with two keys: name, access. The name points to the service as defined in the docker compose
+file.
+
+.. warning::
+
+    This is case sensitive and so the name of the service in the list must be the same name as the service defined.
+
+.. note::
+
+    At this point in time, each x- section has its own pre-defined IAM permissions for services that support IAM access
+    to the resources. In a future version, I might add a configuration file to override that behaviour.
+
+Refer to each x- resource syntax to see which access types are available.
+
+
+Properties
+----------
+
+Unless indicated otherwise, these are the properties for the resource as you would define them using the AWS properties
+in the AWS CloudFormation resource definition.
+
+.. warning::
+
+    In order to update some resources, AWS Sometimes needs to create new ones to replace the once already in place,
+    depending on the type of property you are changing. To do so, AWS will need to have the name of the resource
+    generated, and not set specifically for it. It is a limitation, but in the case of most of the resources, it also
+    allows for continued availability of the service to the resources.
+
+    Therefore, some resources will not be using the `Name` value that you give to it, if you did so.

ecs_composex/compute/SYNTAX.rst

@@ -0,0 +1,48 @@
+ecs_composex.compute
+====================
+
+.. contents::
+
+This module is not strictly a module which the same settings as the other AWS resources. This is a module which allows
+users to create the EC2 compute resources necessary to run the ECS Containers on top of EC2 workloads.
+
+.. note::
+
+    At this point in time, there is no support for creating Capacity providers in CloudFormation, therefore we cannot
+    implement that functionality.
+
+.. note::
+
+    By default, everything is built to use EC2 spot fleet, simply to save money on deployment for testing.
+    Future will allow to run pure OnDemand or hybrid mode.
+
+Define settings in the configs section
+---------------------------------------
+
+At the moment, the settings you can change for the compute definition of your EC2 resources are defined in
+
+configs -> globals -> spot_config
+
+Example:
+
+.. code-block:: yaml
+
+    configs:
+      spot_config:
+        bid_price: 0.42
+        use_spot: true
+        spot_instance_types:
+        m5a.xlarge:
+          weight: 4
+        m5a.2xlarge:
+          weight: 8
+        m5a.4xlarge:
+          weight: 16
+
+With the given AZs of your region, it will create automatically all the overrides to use the spot instances.
+
+.. note::
+
+    This spotfleet comes with a set of predefined Scaling policies, in order to further reduce cost or allow for
+    scaling out based on EC2 metrics.
+

ecs_composex/ecs/SYNTAX.rst

@@ -0,0 +1,106 @@
+ecs_composex.ecs or services
+============================
+
+This is where we try to re-use as much as possible the docker compose (v3) reference as much as possible.
+
+For the definition of the services, you can simply use the already existing Docker compose definition for your services.
+
+However, there are only a limited number of settings that are today working:
+
+* ports
+* environment
+* links
+* depends_on
+
+
+Something you should know
+-------------------------
+
+With the rapid adoption of service discovery and service meshes, by default, all services will be added to an AWS
+CloudMap which is associated with your VPC. If however the CloudMap ID is not provided, they won't be.
+
+I truly believe that using Service Discovery for service to service communication is the way forward.
+
+This really allows to have the same experience on AWS as you would locally in docker compose, only this time, everything
+is further isolated and only explicitly allowed traffic will be allowed.
+
+ECS ComposeX configurations
+---------------------------
+
+This is where developers can leverage the automation implemented in ECS ComposeX to simplify access to their services,
+between services themselves and from external sources too.
+
+
+To define configuration specific to the service and override ECS ComposeX default settings for network configuration,
+you can use the native *configs* key of Docker compose.
+
+.. note::
+
+    To define configuration for your service, simply create a new element/dict in the configs element of the YAML file.
+
+
+ext_sources
+^^^^^^^^^^^
+
+This allows you to define specific ingress control from external sources to your environment. For example, if you have
+to whitelist IP addresses that are to be allowed communication to the services, you can list these, and indicate their
+name which will be shown in the EC2 security group description of the ingress rule.
+
+.. code-block:: yaml
+    configs:
+      app01:
+        ext_sources:
+          - ipv4: 0.0.0.0/0
+            protocol: tcp
+            source_name: all
+          - ipv4: 1.1.1.1/32
+            protocol: icmp
+            source_name: CloudFlareDNS
+
+.. note::
+
+    Future feature is to allow to input a security group ID and the remote account ID to allow ingress traffic from
+    a security group owned by another of your account (or 3rd party).
+
+
+is_public
+^^^^^^^^^
+
+boolean to indicate whether or not the service should be accessible publicly. If set to true, the *load balancer* associated
+to the service will be made public.
+
+use_nlb
+^^^^^^^
+
+Some services will need TCP or UDP based load-balancing. If that is what you need, setting to true will provide your
+service with an NLB to send traffic to your containers.
+
+
+use_alb
+^^^^^^^
+
+Similarly to `use_nlb`_ this however creates an application load-balancer. It will then carry its own Security Group
+and all the permissions for public ingress will be set to the load-balancer security group, where only the ports defined
+on the service will allow ingress from the ALB.
+
+.. warning::
+
+    If you set both use_alb and use_nlb to true, then ALB takes precedence.
+
+use_cloudmap
+^^^^^^^^^^^^
+
+This indicates whether or not you want the service to be added to your VPC CloudMap instance. if set to true, it will
+automatically register the service to the discovery instance.
+
+healthcheck
+^^^^^^^^^^^
+
+At this time, this does not replace the docker compose native functionality of healthcheck. It is a simplified expression of it
+which is used for cloudmap or the load-balancer to register the targets.
+
+.. note::
+
+    This is used for network healthchecks, not service healthcheck
+
+