Deployments should be triggered from ol-home0

[cclauss]

A subtask of #4060

We need to automate the Docker/Python3-based deployment process on ol-home0. We would like to continue using fabfiles for this work but we must shift from process-based deployments to docker-based deployments. We intend to run these services:

• Infobase Move infobase to Python 3 ol-home0 #4059
• solr-updater Create solr-updater docker service AGAIN #4287
• ImportBot Create importbot docker service #4324
• cron jobs Run cron jobs in Docker #4387
• internetarchive/olsystem#110
• Scripts for production deployments Scripts for production deployments #4395

When we do a new deployment, we will need the fabfile to:

1. Checkout the four repos script below or will fabfile.rsync_repo() work as is?:
   1. olsystem
   2. openlibrary
   3. infogami
   4. bookserver-utils (only needed on Web nodes)
2. Build the Docker image and do docker save
3. Run rsync to copy the image to all hosts
   • https://github.com/internetarchive/olsystem/blob/master/bin/olsudo#L25 --> example below
4. docker load and docker up on each of the machines
   • Semi-automatic with commands

Describe the problem that you'd like solved

Proposal & Constraints

Additional context

Stakeholders

[cclauss]

% cat distribute_docker_payload.sh # on ol-home0

#!/bin/bash

FILENAME=${1:-"payload.txt.gz"}

if [[ ! -f $FILENAME ]]; then
    echo "$FILENAME does not exist."
    exit 1
fi
echo "rsyncing $FILENAME to multiple hosts..."

REMOTE_HOSTS="ol-covers0 ol-web1 ol-web2"
for REMOTE_HOST in $REMOTE_HOSTS
do
    echo "Starting rsync to $REMOTE_HOST..."
    rsync -a --delete --no-owner --no-group --verbose $FILENAME "$REMOTE_HOST:/home/cclauss/"
    echo "Finished rsync to $REMOTE_HOST..."
done

[cclauss]

% cat scripts/update_ol_repos.sh

#!/bin/bash

# This deployment script is used to update the core repos on an ol-server node.
# The core Open Library repos are:
# 1. https://github.com/internetarchive/olsystem
# 2. https://git.archive.org/jake/booklending_utils
# 3. https://github.com/internetarchive/openlibrary
# 4. https://github.com/internetarchive/infogami

# This script takes a few environment variables:
# CAUTION: To git clone olsystem, environment variables must be set...
# Set $GITHUB_USERNAME or $USER will be used.
# Set $GITHUB_TOKEN or this script will halt.
if [[ -z ${GITHUB_TOKEN} ]]; then
    echo "FATAL: Can not git clone olsystem" ;
    exit 1 ;
fi

cd /opt/olsystem                    && sudo git pull origin master
cd /opt/booklending_utils           && sudo git pull origin master
cd /opt/openlibrary                 && sudo git pull origin master
cd /opt/openlibrary/vendor/infogami && sudo git pull origin master
cd /opt/openlibrary

[cclauss]

Does fabfile.rsync_repo() work as is? Or will we need a script like the one above that we run on each node?

[cclauss]

Closing in favor of #4423 because we have too many tracking issues.