Organizational and maintenance scripts for Find Work repositories
This repository is heavily based on twolfson/twolfson.com-scripts.
We maintain all of our service in a single VM via Vagrant. We use vagrant-lxc to get container performance while maintaining the ease of using a VM.
We choose to use Vagrant over Docker for a few reasons:
- Vagrant shares more common knowledge with a production server (i.e. use
ssh
) - We are more comfortable with VMs and don't want to drain time debugging Docker
- This includes edge cases like architectural structure (e.g. placing PostgreSQl in same or separate containers, minimizing downtime on container restart)
To get our server running, perform the following steps:
- Install Vagrant via its download page
- Run the following shell commands
# Clone our repository
git clone git@github.com:findworkco/scripts.git scripts
cd scripts
# Install our plugin dependencies
# https://github.com/fgrehm/vagrant-lxc/tree/v1.2.1#requirements
sudo apt-get install lxc redir
# Install our Vagrant plugin
vagrant plugin install vagrant-lxc
# Set up `sudoers` bindings for `vagrant-lxc` (avoids `sudo` for `vagrant up`/`vagrant ssh`/etc)
# https://github.com/fgrehm/vagrant-lxc/blob/v1.2.1/lib/vagrant-lxc/command/sudoers.rb
vagrant lxc sudoers
# Decrypt our configuration
CONFIG_COPY_ONLY=TRUE bin/decrypt-config.sh
# To decrypt our secrets (e.g. production db password), use
# CONFIG_COPY_ONLY=FALSE bin/decrypt-config.sh
# Start our Vagrant instance
vagrant up
# SSH into the machine
vagrant ssh
We manage our port reservations via docs/ports.md.
If you are registering a new service, please update the document via a pull request.
This repository has the following file structure:
.bundle/
- Configuration for Bundler (used for managing Ruby gems)bin/
- Contains our executable files (e.g. deployment scripts)data/
- Contains static files used during provisioning- This starts at
/
as if it were the root of a file system - For multiple environment projects, it's good to have a
data/{{env}}
for each setup (e.g.data/development
,data/production
)
- This starts at
src/
- Container for our bootstrapping scriptstest/
- Container for our test filesREADME.md
- Documentation for this repositoryVagrantfile
- Configuration for Vagrant
To provision a new server via Digital Ocean, follow the steps below:
- If we don't have a Digital Ocean SSH key pair yet, then generate one
- Create a new Ubuntu based droplet with our SSH key (14.04 x64)
- Add public key to data/home/ubuntu/.ssh/authorized_keys so we can
ssh
into theubuntu
user- Digital Ocean's SSH key will initially be registered to
root
user but we dislike having direct SSH access into aroot
user
- Digital Ocean's SSH key will initially be registered to
- Once droplet has started, set up our
~/.ssh/config
on the local machine
# Replace `digital-my-server` with a better name
# Replace 127.0.0.1 with droplet's public IP
Host digital-my-server
User root
HostName 127.0.0.1
- Install our SSL certificates
# Install certbot for NGINX # DEV: We run all of this by hand as this only works for 1 server so far. We'll need to solve multiple server soon # Taken from: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04 # Verified in: https://certbot.eff.org/lets-encrypt/ubuntutyakkety-nginx sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install python-certbot-nginx # Obtain our certificates for NGINX and explicitly our domain sudo certbot --nginx -d findwork.co -d www.findwork.co # Email address: todd@findwork.co # Terms of Service: Agree # Share email with EFF: Yes # No redirect/redirect: No redirect (already heavily configured) # Files are stored at /etc/letsencrypt/live/findwork.co/fullchain.pem # Outside of SSH, verify the certificate looks good # https://www.ssllabs.com/ssltest/analyze.html?d=findwork.co # Verify renewal works as expected sudo certbot renew --dry-run
- Install our Diffie-Hellman group to the server
bin/install-nginx-data-remote.sh digital-my-server --dhparam path/to/dhparam.pem
- If you are trying to get a replica working (e.g. don't have these certificates), then self-signed certificates and a
dhparam.pem
can be generated via theopenssl
commands inbin/bootstrap-vagrant.sh
- Install our PGP private key to the server
bin/install-pgp-data-remote.sh digital-my-server --secret-key path/to/private.rsa
- If you don't have the
private.rsa
file on hand, it can be dumped via- Find full fingerprint of key we want to export
gpg --fingerprint
- Fingerprint will be
740D DBFA...
inKey fingerprint = 740D DBFA...
- Extract private key to file
gpg --export-secret-keys --armor {{fingerprint}} > private.rsa
--armor
exports a human-friendly ASCII format instead of binary
- Find full fingerprint of key we want to export
- If you are trying to get a replica working (e.g. don't have these certificates), then a key can be generated via these instructions
- Install our server configuration
bin/install-config-data-remote.sh digital-my-server
- If you are trying to get a replica working (e.g. don't have these certificates), then use
CONFIG_COPY_ONLY=TRUE
to prevent SOPS errors
- Bootstrap our server
bin/bootstrap-remote.sh digital-my-server
- Update
~/.ssh/config
to useUser ubuntu
instead ofUser root
- During the bootstrap process, we intentionally lock our
root
access viassh
for security
- During the bootstrap process, we intentionally lock our
- Run our tests on the server
bin/test-remote.sh digital-my-server
- Add swap space to the server (necessary for memory-intensive installs)
- https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
- On a 1GB Digial Ocean instance, we use:
sudo dd if=/dev/zero of=/swapfile bs=512M count=4
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo nano /etc/fstab
sudo sysctl vm.swappiness=10
sudo nano /etc/sysctl.conf
sudo sysctl vm.vfs_cache_pressure=50
sudo nano /etc/sysctl.conf
- Install Librato via https://metrics.librato.com/integrations
- Instead of using
curl | sudo bash
. Download the script viawget
, verify it looks good vialess
, and run it viasudo ./{{script-name}}
- Instead of using
We reuse our provisioning script for managing server state. As a result, we can reuse it for updates:
bin/bootstrap-remote.sh digital-my-server
# If we need to use a non-master ref, then pass it as a second parameter
# bin/bootstrap-remote.com.sh digital-my-server dev/new.feature
To deploy a service, use its respective bin/deploy-*.sh
script. Here's an example with findworkco/app
:
bin/deploy-app.sh digital-my-server
# If we need to deploy a non-master ref, then pass as a second parameter
# bin/deploy-app.sh digital-my-server dev/new.feature
We maintain a set of secrets (e.g. passwords) for provisioning in production in config/static-secrets.enc.yml
. To edit these files locally, perform the following steps:
- Install SOPS' dependencies as specified by https://github.com/mozilla/sops/tree/0494bc41911bc6e050ddd8a5da2bbb071a79a5b7#up-and-running-in-60-seconds
- Install our consistent patched SOPS version
pip install --upgrade sops==1.15
- Ask a coworker for the
findworkco/scripts
PGP private key- We assume you will receive it as
private.rsa
- For coworkers, see the Provisioning a new server section for dump commands
- We assume you will receive it as
- Install the
findworkco/scripts
PGP private key to GPGgpg --import private.rsa
- Edit the SOPS file
bin/edit-secrets.sh
If you would like to learn more about PGP and SOPS, @twolfson has prepared this document:
https://gist.github.com/twolfson/01d515258eef8bdbda4f
We try to keep our services as secure as possible via the following means:
- Restricting shell access for SSH users
- Preventing password only authentication for SSH
- Restricting permissions on sensitive files (e.g. SSL certificates, NGINX configurations)
We have gone out of our way to patch the following CVE's:
- Shellshock - Patched by upgrading bash (default is fine on Ubuntu 14.04)
- Heartbleed - Patched by upgrading NGINX (default is fine on Ubuntu 14.04)
- POODLE - Patched by restricting SSL methods used by NGINX
- Logjam - Patched by using new Diffie-Hellman group
We use [Serverspec][] for testing local and remote servers. This is a Ruby gem so you will need it installed to run our tests:
# Install bundler to manage gems for local directory
gem install bundler
# Install dependencies for this repo
bundle install
# Run our tests
./test.sh
To make iterating on our test suite faster, we have set up SKIP_LINT
and SKIP_PROVISION
environment variables. This skips running linting and vagrant provision
in our tests:
# Skip both linting and provisioning
SKIP_LINT=TRUE SKIP_PROVISION=TRUE ./test.sh
To run our test suite against a production machine, we can use the bin/test-remote.sh
script.
bin/test-remote.sh digital-my-server
All rights reserved, Shoulders of Titans LLC