Feature x-docdb (#252)

* Implemented DocDB
* Refactored x-rds functions as DocDB and RDS are following the same pattern

README.rst

@@ -112,8 +112,9 @@ AWS Services
     * Your mesh made easy, backed up by AWS CloudMap for services discovery.
 * `AWS SQS`_: queues for distributed workloads
     * With AutoScaling support based on number of messages
-* `AWS RDS`_: databases integration made easy
-* `AWS DynamoDB`_: Create and use dynamodb tables with your services
+* `AWS RDS`_
+* `AWS DynamoDB`_
+* `AWS DocumentDB`_
 * `AWS S3`_: Buckets creation and access simplified
     * On import via **Lookup**, automatically identifies KMS encryption key
 * `AWS IAM`_
@@ -265,6 +266,7 @@ This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypack
 .. _AWS S3: https://nightly.docs.ecs-composex.lambda-my-aws.io/syntax/composex/s3.html
 .. _AWS IAM: https://nightly.docs.ecs-composex.lambda-my-aws.io/syntax/composex/ecs.details/iam.html
 .. _AWS SNS: https://nightly.docs.ecs-composex.lambda-my-aws.io/syntax/composex/sns.html
+.. _AWS DocumentDB: https://nightly.docs.ecs-composex.lambda-my-aws.io/syntax/composex/docdb.html
 
 .. _AWS EC2: https://nightly.docs.ecs-composex.lambda-my-aws.io/features.html#ec2-resources-for-ecs-cluster
 .. _AWS AppMesh: https://nightly.docs.ecs-composex.lambda-my-aws.io/readme/appmesh.html

docs/features.rst

@@ -9,6 +9,7 @@
     features/appmesh
     features/rds
     features/dyndb
+    features/docdb
     features/s3
     features/sqs
     features/kms

docs/features/docdb.rst

@@ -0,0 +1 @@
+.. include:: ../../ecs_composex/docdb/README.rst

docs/modules_syntax.rst

@@ -18,6 +18,7 @@
     syntax/composex/elbv2
     syntax/composex/rds
     syntax/composex/dynamodb
+    syntax/composex/docdb
     syntax/composex/s3
     syntax/composex/sqs
     syntax/composex/sns

docs/syntax/composex/docdb.rst

@@ -0,0 +1,131 @@
+.. _docdb_syntax_reference:
+
+=========
+x-docdb
+=========
+
+Syntax
+=======
+
+.. code-block:: yaml
+
+    x-docdb:
+      docdb-01:
+        Properties: {}
+        Settings: {}
+        Services: []
+        Lookup: {}
+        MacroParameters: {}
+
+Properties
+===========
+
+DocDB Cluster is rather very simple in its configuration. There aren't 200+ combinations of EngineName and Engine Version
+as for RDS, make life very easy.
+
+Howerver you can copy-paste all the properties you would find in the `DocDB Cluster properties`_, some properties will be
+ignored in order to keep the automation going:
+
+* MasterUsername and MasterUserPassword
+    These two will be auto generated and stored in secrets manager. The services linked to it will be granted **GetSecretValue** to it.
+
+* VpcSecurityGroupIds
+    The security group will be generated for the DB specifically and allow services listed only.
+
+* AvailabilityZones
+    Under trial, but not sure given that we give a Subnet Group why one would also define the AZs and it might conflict.
+
+* DBClusterIdentifier
+    As usual, named resources make for a nightmare to rename etc. Instead, there will be a **Name** tag associated with your Cluster.
+
+* DBSubnetGroupName
+    Equally gets created only. For now.
+
+* SnapshotIdentifier
+    Untested - 2020-11-13 - will support it later.
+
+
+Services
+========
+
+The syntax for listing the services remains the same as the other x- resources.
+
+.. code-block:: yaml
+
+    Services:
+      - name: <service/family name>
+        access: <str>
+
+Access types
+------------
+
+.. warning::
+
+    The access key value do not have an effect at this stage.
+
+Settings
+========
+
+The only setting for DocumentDB is **EnvNames** as for every other resources.
+
+.. hint::
+
+    Given that the DB Secret attachment populates host, port etc., we expose as env vars the **Secret** associated to the DB,
+    not the DB itself.
+
+MacroParameters
+================
+
+These parameters will allow you to define extra parameters to define your cluster successfully.
+In the future you should be able to define your `DocDB Cluster Parameters`_ there.
+
+Instances
+---------
+
+List of DocDB instances. The aspiration is to follow the same syntax as the `DocDB Instance`_.
+
+.. note::
+
+    Not all Properties are respected, instead, they follow logically the attachment to the DocDB Cluster.
+
+
+.. code-block:: yaml
+
+    Instances:
+      - DBInstanceClass: <db instance type>
+        PreferredMaintenanceWindow: <window definition>
+        AutoMinorVersionUpgrade: bool
+
+
+.. hint::
+
+    If you do not define an instance, ECS ComposeX automatically creates a new one with a single node of type **db.t3.medium**
+
+
+Lookup
+========
+
+Lookup for Document DB is available!
+
+.. warning::
+
+    For some reason the group resource tag API returns two different clusters even though they are the same one.
+    Make sure to specify the *Name* along with Tags until we figure an alternative solution.
+    Sorry for the inconvenience.
+
+
+Examples
+========
+
+.. literalinclude:: ../../../use-cases/docdb/create_only.yml
+    :language: yaml
+    :caption: Sample to crate two DBs with different instances configuration
+
+.. literalinclude:: ../../../use-cases/docdb/create_lookup.yml
+    :language: yaml
+    :caption: Create a DocDB and import an existing one.
+
+
+.. _DocDB Cluster properties: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-docdb-dbcluster.html
+.. _DocDB Instance: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-docdb-dbinstance.html
+.. _DocDB Cluster Parameters: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-docdb-dbclusterparametergroup.html

ecs_composex/common/compose_resources.py

@@ -84,6 +84,9 @@ def __init__(self, name, definition, settings):
             if not keyisset("Settings", self.definition)
             else self.definition["Settings"]
         )
+        self.use = (
+            None if not keyisset("Use", self.definition) else self.definition["Use"]
+        )
         self.lookup = (
             None
             if not keyisset("Lookup", self.definition)
@@ -102,8 +105,10 @@ def __init__(self, name, definition, settings):
             if not keyisset("Services", self.definition)
             else self.definition["Services"]
         )
-        self.use = (
-            None if not keyisset("Use", self.definition) else self.definition["Use"]
+        self.parameters = (
+            {}
+            if not keyisset("MacroParameters", self.definition)
+            else self.definition["MacroParameters"]
         )
         self.cfn_resource = None
         self.output_properties = {}
@@ -268,15 +273,16 @@ def generate_outputs(self):
         """
         for output_prop_name in self.output_properties:
             definition = self.output_properties[output_prop_name]
-            if definition[1] is Ref and definition[2] is None:
-                value = Ref(self.cfn_resource)
-            elif definition[1] is Ref and definition[2] is not None:
-                value = Ref(definition[2])
-            elif definition[1] is GetAtt and isinstance(definition[2], str):
-                value = GetAtt(self.cfn_resource, definition[2])
+            if definition[2] is Ref:
+                value = Ref(definition[1])
+            elif definition[2] is GetAtt:
+                value = GetAtt(definition[1], definition[3])
             else:
-                raise ValueError(
-                    f"Something was not defined properly for output properties of {self.logical_name}"
+                raise TypeError(
+                    f"3rd argument for {definition[0]} must be one of",
+                    (Ref, GetAtt),
+                    "Got",
+                    definition[2],
                 )
             self.outputs.append(Output(NONALPHANUM.sub("", definition[0]), Value=value))
 

ecs_composex/docdb/README.rst

@@ -0,0 +1,39 @@
+.. _aws_docdb_readme:
+
+=======================
+AWS DocumentDB (DocDB)
+=======================
+
+DocumentDB is yet another great implementation of a popular service (MongoDB) made as a service with the harvest power
+of AWS Storage.
+
+Similar in many many ways to RDS Aurora clusters with MySQL and PostgreSQL compatibility, this time, for MongoDB.
+
+Properties
+==========
+
+AWS DocDB Cluster configuration is a lot simplier than its SQL cousins. With very minimum properties, you can, using ComposeX,
+deploy clusters and automatically link these to your services.
+
+.. tip::
+
+    For production workloads, to avoid any CFN deadlock situations, I recommend you generate the CFN templates for docdb,
+    and deploy the stacks separately. Using Lookup you can use existing DocDB clusters with your new services.
+
+
+Credentials
+===========
+
+The credentials strucutre remains the same as for RDS SQL versions
+
+.. code-block:: json
+    :caption: DocumentDB secret structure after attachment
+
+    {
+      "dbClusterIdentifier": "<str>",
+      "password": "<str>",
+      "engine": "<str>",
+      "port": <int>,
+      "host": "<str>",
+      "username": "<str>"
+    }

ecs_composex/docdb/__init__.py

@@ -0,0 +1,16 @@
+#  -*- coding: utf-8 -*-
+#   ECS ComposeX <https://github.com/lambda-my-aws/ecs_composex>
+#   Copyright (C) 2020  John Mille <john@lambda-my-aws.io>
+#  #
+#   This program is free software: you can redistribute it and/or modify
+#   it under the terms of the GNU General Public License as published by
+#   the Free Software Foundation, either version 3 of the License, or
+#   (at your option) any later version.
+#  #
+#   This program is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#   GNU General Public License for more details.
+#  #
+#   You should have received a copy of the GNU General Public License
+#   along with this program.  If not, see <http://www.gnu.org/licenses/>.

ecs_composex/docdb/docdb_ecs.py

@@ -0,0 +1,98 @@
+#  -*- coding: utf-8 -*-
+#   ECS ComposeX <https://github.com/lambda-my-aws/ecs_composex>
+#   Copyright (C) 2020  John Mille <john@lambda-my-aws.io>
+#  #
+#   This program is free software: you can redistribute it and/or modify
+#   it under the terms of the GNU General Public License as published by
+#   the Free Software Foundation, either version 3 of the License, or
+#   (at your option) any later version.
+#  #
+#   This program is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#   GNU General Public License for more details.
+#  #
+#   You should have received a copy of the GNU General Public License
+#   along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+Module to link DocDB cluster to ECS Services.
+"""
+
+from ecs_composex.common import keyisset
+from ecs_composex.common import LOG
+from ecs_composex.docdb.docdb_params import DOCDB_PORT
+from ecs_composex.tcp_resources_settings import handle_new_tcp_resource
+from ecs_composex.rds.rds_ecs import (
+    import_dbs,
+    lookup_rds_resource,
+    validate_rds_lookup,
+    DB_SECRET_T,
+)
+
+
+def create_docdb_cluster_config_mapping(resource, db_config):
+    """
+
+    :param resource:
+    :param db_config:
+    :return:
+    """
+    mapping = {
+        resource.logical_name: {
+            "VpcSecurityGroupIds": [
+                k["VpcSecurityGroupId"]
+                for k in db_config["VpcSecurityGroups"]
+                if k["Status"] == "active"
+            ],
+            "Port": db_config["Port"],
+            resource.logical_name: db_config["DBClusterIdentifier"],
+        }
+    }
+    if keyisset(DB_SECRET_T, db_config):
+        mapping[resource.logical_name][DB_SECRET_T] = db_config[DB_SECRET_T]
+    return mapping
+
+
+def create_lookup_mappings(mappings, lookup_dbs, settings):
+    """
+    Function to create the DocumentDB mappings to add to services templates
+
+    :param dict mappings:
+    :param list lookup_dbs:
+    :param ecs_composex.common.settings.ComposeXSettings settings: The settings for ComposeX Execution
+    """
+    for db in lookup_dbs:
+        validate_rds_lookup(db.name, db.lookup)
+        db_config = lookup_rds_resource(db.lookup, settings.session)
+        if not db_config:
+            LOG.warn(
+                f"No RDS DB Configuration could be defined from provided lookup. Skipping {db.name}"
+            )
+            return
+        config = create_docdb_cluster_config_mapping(db, db_config)
+        mappings.update(config)
+
+
+def docdb_to_ecs(resources, services_stack, res_root_stack, settings):
+    """
+    Entrypoint function to map new and lookup resources to ECS Services
+
+    :param list resources:
+    :param ecs_composex.common.stacks.ComposeXStack services_stack:
+    :param ecs_composex.common.stacks.ComposeXStack res_root_stack:
+    :param ecs_composex.common.settings.ComposeXSettings settings:
+    """
+    db_mappings = {}
+    new_resources = [
+        resources[res_name] for res_name in resources if not resources[res_name].lookup
+    ]
+    lookup_resources = [
+        resources[res_name] for res_name in resources if resources[res_name].lookup
+    ]
+    for new_res in new_resources:
+        handle_new_tcp_resource(new_res, res_root_stack, DOCDB_PORT)
+    create_lookup_mappings(db_mappings, lookup_resources, settings)
+    for lookup_res in lookup_resources:
+        if keyisset(lookup_res.logical_name, db_mappings):
+            import_dbs(lookup_res, db_mappings, mapping_name="DocDb")