Skip to content

Latest commit

 

History

History
84 lines (56 loc) · 3.85 KB

aws.md

File metadata and controls

84 lines (56 loc) · 3.85 KB
Raw

How to Release the AI-Lab

  1. Take care to update latest file doc/changes/changes_*.md
    • Correct version number
    • Current date
    • Code name
    • Summary
    • Remove sections without tickets or add n/a
  2. Push and merge your current Pull-request
  3. Run release-droid
  4. Edit draft release on GitHub and make it final

Release-droid command line:

java -jar ~/path/to/release-droid-*.jar \
     -n ai-lab -goal release \
     -guide /tmp/release-guide.html

AWS Build and Release Workflow

The following diagram shows the high-level steps to generate the images:

image info

Setup EC2

Creates an EC2 instance based on an Ubuntu AMI via AWS Cloudformation. It also creates a new keypair dynamically. After the export has finished, the cloudformation stack and the keypair is removed.

Install

Installs all dependencies via Ansible:

  • installs and configures Jupyter
  • installs Docker and adds the user jupyter to the docker group
  • changes the netplan configuration. This is necessary to have proper network configuration when running the VM image

Finally, the default password will be set, and also the password will be marked as expired, such that the user will be forced to enter a new password during initial login. Also, the ssh password authentication will be enabled, and for security reasons the folder "~/.ssh" will be removed.

Export

The export creates an AMI based on the running EC2 instance and exports the AMI as VM image in the default formats to a S3 bucket.

Release

The release is executed in a AWS Codebuild job, the following diagram shows the flow.

image info

AWS S3 Bucket

The bucket has private access. In order to control access, the Bucket cloudformation stack also contains a Cloudfront distribution. Public Https access is only possibly through Cloudfront. Another stack contains a Web application firewall (WAF), which will be used by the Cloudfront distribution. Due to restrictions in AWS, the WAF stack needs to be deployed in region "us-east-1". The WAF stack provides two rules which aim to minimize a possible bot attack:

Name Explanation Priority
VMBucketRateLimit Declares the minimum possible rate limit for access: 100 requests in a 5 min interval. 0
CAPTCHA Forces a captcha action for any IP which does not match a predefined set of IP-addresses. 1

Involved Cloudformation stacks

The following diagram shows the involved cloudformation stacks: image info

The following resources are permanent and need to be deployed using the "deploy" commands:

  • DATA-SCIENCE-SANDBOX-VM-Bucket
  • DATA-SCIENCE-SANDBOX-CI-TEST-CODEBUILD
  • DATA-SCIENCE-SANDBOX-RELEASE-CODEBUILD

The EC2-stack lives only during the creation of a new sandbox image.

Tagging AWS Resources

Each of the involved resources might cause costs: cloudformation stacks, AMI, EC2 key-pairs.

To enable keeping track of all these resources, the implementation tags them after creation with a specific keyword (called asset-id).

The S3 objects are identified by the prefix in the S3 bucket.

The command tags only the dynamically created entities with the asset-id but not the permanent cloudformation stacks.

The command show-aws-assets lists all assets which were created during the execution.

  • This is very useful if an error occured.
  • If the creation of a sandbox finished normally the list should contain only the AMI, images (S3 objects) and the export tasks (one for each image).