Skip to content

Keypair Rotation

Jump to bottom
Aaron D Borden edited this page Oct 23, 2020 · 25 revisions

This document describes what secret credentials we maintain and how to rotate them.

Ansible Vault

We currently have two Ansible Vaults: production and sandbox, each with their own vault password.

You'll need both the existing key and the new key. Adjust the parameters for the new secret and run ansible rekey in datagov-deploy. Don't forget to add the new key to Google Drive.

Vault re-keys should be done as a hotfix to minimize the chance of merge conflicts within vaults and so all branches (master and develop) aren't using multiple key versions.

For production, you'll want to rekey all the vault files for production, staging, and mgmt inventories (i.e. filter out the sandbox vault files).

$ vault_id=production
$ vault_password_file=.secrets/production-v$(date +%Y%m%d).txt
$ pwgen -s 64 1 > ${vault_password_file}  # Generate a random 64-character string and put it in the key file.
$ find . -name vault.yml | grep -v 'sandbox\|local' | xargs pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file}

To rekey the sandbox vault:

$ vault_id=sandbox
$ vault_password_file=.secrets/sandbox-v$(date +%Y%m%d).txt
$ pwgen -s 64 1 > ${vault_password_file}  # Generate a random 64-character string and put it in the key file.
$ find . -name vault.yml | grep 'sandbox\|local' | xargs pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file}

Before merging, give a heads up to other developers. Any in-progress vault changes will need to be rebased in order to pick up the new key by the author.

Update deployment environments

Immediately after merging, all environments must be updated with the new secret.

For each environment, update the jumpbox with the new key. Add a new file for the key (e.g. /etc/datagov/sandbox-v20200709.txt) with permissions root:operators 0640. Update the symlink to point to the new key file.

$ ln -snf sandbox-v20200709.txt /etc/datagov/ansible-secret.txt

Validate the change with ansible and remove all old keys.

$ pipenv run ansible -m ping all

Finally, update the ansible-vault credential in Jenkins.

SSH root keys

The root ssh keys allow access to the ubuntu user. These keys are only used for the initial provision of the jumpbox. After that, operators use their personal keys to access all hosts.

  1. Generate a new SSH key. When prompted, set a 16+ character password. For the comment, use the key name (datagov-environment-vX).

    $ ssh-keygen -f ~/.ssh/datagov-environment-vX

  2. Upload the private and public keys to the shared Drive. Share the password with teammates over a secure channel.

  3. For BSP environments (production and staging), create a Service Now ticket to update the SSH key in the CloudFormation templates. Attach the public key to the ticket. BSP does not need the private key.

  4. For AWS Sandbox environments, import the key in the AWS Console. Update the keypair name in datagov-infrastructure-live.

Clone this wiki locally