Skip to content
This repository has been archived by the owner on Dec 25, 2019. It is now read-only.
/ scripts Public archive

Organizational and maintenance scripts for Find Work repositories

1 star 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

findworkco/scripts

BranchesTags

Repository files navigation

scripts wercker status

Organizational and maintenance scripts for Find Work repositories

This repository is heavily based on twolfson/twolfson.com-scripts.

Background

We maintain all of our service in a single VM via Vagrant. We use vagrant-lxc to get container performance while maintaining the ease of using a VM.

We choose to use Vagrant over Docker for a few reasons:

  • Vagrant shares more common knowledge with a production server (i.e. use ssh)
  • We are more comfortable with VMs and don't want to drain time debugging Docker
    • This includes edge cases like architectural structure (e.g. placing PostgreSQl in same or separate containers, minimizing downtime on container restart)

Getting Started

To get our server running, perform the following steps:

# Clone our repository
git clone git@github.com:findworkco/scripts.git scripts
cd scripts

# Install our plugin dependencies
# https://github.com/fgrehm/vagrant-lxc/tree/v1.2.1#requirements
sudo apt-get install lxc redir

# Install our Vagrant plugin
vagrant plugin install vagrant-lxc

# Set up `sudoers` bindings for `vagrant-lxc` (avoids `sudo` for `vagrant up`/`vagrant ssh`/etc)
# https://github.com/fgrehm/vagrant-lxc/blob/v1.2.1/lib/vagrant-lxc/command/sudoers.rb
vagrant lxc sudoers

# Decrypt our configuration
CONFIG_COPY_ONLY=TRUE bin/decrypt-config.sh
# To decrypt our secrets (e.g. production db password), use
#   CONFIG_COPY_ONLY=FALSE bin/decrypt-config.sh

# Start our Vagrant instance
vagrant up

# SSH into the machine
vagrant ssh

Documentation

Ports

We manage our port reservations via docs/ports.md.

If you are registering a new service, please update the document via a pull request.

File structure

This repository has the following file structure:

  • .bundle/ - Configuration for Bundler (used for managing Ruby gems)
  • bin/ - Contains our executable files (e.g. deployment scripts)
  • data/ - Contains static files used during provisioning
    • This starts at / as if it were the root of a file system
    • For multiple environment projects, it's good to have a data/{{env}} for each setup (e.g. data/development, data/production)
  • src/ - Container for our bootstrapping scripts
  • test/ - Container for our test files
  • README.md - Documentation for this repository
  • Vagrantfile - Configuration for Vagrant

Provisioning a new server

To provision a new server via Digital Ocean, follow the steps below:

  • If we don't have a Digital Ocean SSH key pair yet, then generate one
  • Create a new Ubuntu based droplet with our SSH key (14.04 x64)
  • Add public key to data/home/ubuntu/.ssh/authorized_keys so we can ssh into the ubuntu user
    • Digital Ocean's SSH key will initially be registered to root user but we dislike having direct SSH access into a root user
  • Once droplet has started, set up our ~/.ssh/config on the local machine
# Replace `digital-my-server` with a better name
# Replace 127.0.0.1 with droplet's public IP
Host digital-my-server
    User root
    HostName 127.0.0.1
  • Install our SSL certificates 
    # Install certbot for NGINX
# DEV: We run all of this by hand as this only works for 1 server so far. We'll need to solve multiple server soon
# Taken from: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04
# Verified in: https://certbot.eff.org/lets-encrypt/ubuntutyakkety-nginx
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx

# Obtain our certificates for NGINX and explicitly our domain
sudo certbot --nginx -d findwork.co -d www.findwork.co
# Email address: todd@findwork.co
# Terms of Service: Agree
# Share email with EFF: Yes
# No redirect/redirect: No redirect (already heavily configured)
# Files are stored at /etc/letsencrypt/live/findwork.co/fullchain.pem

# Outside of SSH, verify the certificate looks good
# https://www.ssllabs.com/ssltest/analyze.html?d=findwork.co

# Verify renewal works as expected
sudo certbot renew --dry-run
  • Install our Diffie-Hellman group to the server
    • bin/install-nginx-data-remote.sh digital-my-server --dhparam path/to/dhparam.pem
    • If you are trying to get a replica working (e.g. don't have these certificates), then self-signed certificates and a dhparam.pem can be generated via the openssl commands in bin/bootstrap-vagrant.sh
  • Install our PGP private key to the server
    • bin/install-pgp-data-remote.sh digital-my-server --secret-key path/to/private.rsa
    • If you don't have the private.rsa file on hand, it can be dumped via
      • Find full fingerprint of key we want to export
        • gpg --fingerprint
        • Fingerprint will be 740D DBFA... in Key fingerprint = 740D DBFA...
      • Extract private key to file
        • gpg --export-secret-keys --armor {{fingerprint}} > private.rsa
        • --armor exports a human-friendly ASCII format instead of binary
    • If you are trying to get a replica working (e.g. don't have these certificates), then a key can be generated via these instructions
  • Install our server configuration
    • bin/install-config-data-remote.sh digital-my-server
    • If you are trying to get a replica working (e.g. don't have these certificates), then use CONFIG_COPY_ONLY=TRUE to prevent SOPS errors
  • Bootstrap our server
    • bin/bootstrap-remote.sh digital-my-server
  • Update ~/.ssh/config to use User ubuntu instead of User root
    • During the bootstrap process, we intentionally lock our root access via ssh for security
  • Run our tests on the server
    • bin/test-remote.sh digital-my-server
  • Add swap space to the server (necessary for memory-intensive installs)
  • Install Librato via https://metrics.librato.com/integrations
    • Instead of using curl | sudo bash. Download the script via wget, verify it looks good via less, and run it via sudo ./{{script-name}}

Updating a server configuration

We reuse our provisioning script for managing server state. As a result, we can reuse it for updates:

bin/bootstrap-remote.sh digital-my-server

# If we need to use a non-master ref, then pass it as a second parameter
# bin/bootstrap-remote.com.sh digital-my-server dev/new.feature

Deploying a service

To deploy a service, use its respective bin/deploy-*.sh script. Here's an example with findworkco/app:

bin/deploy-app.sh digital-my-server

# If we need to deploy a non-master ref, then pass as a second parameter
# bin/deploy-app.sh digital-my-server dev/new.feature

Editing secrets

We maintain a set of secrets (e.g. passwords) for provisioning in production in config/static-secrets.enc.yml. To edit these files locally, perform the following steps:

If you would like to learn more about PGP and SOPS, @twolfson has prepared this document:

https://gist.github.com/twolfson/01d515258eef8bdbda4f

Security

We try to keep our services as secure as possible via the following means:

  • Restricting shell access for SSH users
  • Preventing password only authentication for SSH
  • Restricting permissions on sensitive files (e.g. SSL certificates, NGINX configurations)

Patched major CVE's

We have gone out of our way to patch the following CVE's:

Testing

We use [Serverspec][] for testing local and remote servers. This is a Ruby gem so you will need it installed to run our tests:

# Install bundler to manage gems for local directory
gem install bundler

# Install dependencies for this repo
bundle install

# Run our tests
./test.sh

To make iterating on our test suite faster, we have set up SKIP_LINT and SKIP_PROVISION environment variables. This skips running linting and vagrant provision in our tests:

# Skip both linting and provisioning
SKIP_LINT=TRUE SKIP_PROVISION=TRUE ./test.sh

Validating production

To run our test suite against a production machine, we can use the bin/test-remote.sh script.

bin/test-remote.sh digital-my-server

Copyright

All rights reserved, Shoulders of Titans LLC

About

Organizational and maintenance scripts for Find Work repositories

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages