Skip to content

1. Manage EC2 Clusters

Jump to bottom
JOHN THORPE edited this page Apr 2, 2021 · 1 revision

Chapter 1. Manage AWS EC2 Clusters

Previous page: Home | Next page: 2. Build Dorylus | Home: Home

Our system can be built upon AWS EC2 machines. If your instances are accidentally missing on the cloud console, check the "region option" on the upper-right corner of the console. You can only view your resources within the correct region.

Our system relies on 2 different roles of machines:

  1. Multiple graph servers (data servers)
  2. One or more weight servers

All these machines should be under the same ARN, and within the same AWS service region. You should have password-less SSH connection configured for logging in all your machines remotely.

The ec2man Python module alone is general: If you wanna use it for other purposes, we do not limit the context names.

Install & Configure AWS CLI

You should first get AWS CLI on your local shell, in order to remotely manage your AWS resources & services.

On Ubuntu:

Local$ sudo apt install awscli

On ArchLinux:

Local$ sudo pacman -S aws-cli

On MacOS:

Local$ brew install awscli

Configure AWS variables (You must specify the region here, instead of leaving it blank):

Local$ aws configure
...     # Enter the required fields

The above command will set the configurations you entered as the default config profile. You can also add more custom profiles manually (here the example name is josehu):

Local$ vim ~/.aws/credentials

[default]
aws_access_key_id = ...
aws_secret_access_key = ...

[josehu]
aws_access_key_id = ...
aws_secret_access_key = ...

Local$ vim ~/.aws/config

[default]
region = us-east-2

[profile josehu]
region = us-east-1

Setup the ec2man Python Module

Ensure you have python3 with version >= 3.5. Install AWS client module boto3:

Local$ pip3 install boto3

Create a text file named profile under ec2man/, which specifies your AWS ARN string and the CLI profile to use:

Local$ vim ec2man/profile

default                     # AWS profile to use
dorylus                     # User name on cluster
/home/dorylus/.ssh/id_rsa   # SSH key for accessing cluster
us-east-2                   # AWS region

Create a text file named machines under ec2man/, which contains the required info for all the machines you are using to build up the system. Though this ec2man is general and does not constrain the context (role) names of your clusters, for Dorylus system you should categorize your machines into 2 different contexts graph and weight. An optional third context is nfs if you want to use an NFS server to hold the data (this is recommended).

Local$ vim ec2man/machines

i-1234567890abcdefg     graph
i-1234567890abcdefh     graph
i-1234567890abcdefi     weight
i-1234567890abcdefj     weight
i-1234567890abcdefk     nfs     ## If using NFS to hold data

Finally run the setup command:

Local$ python3 -m ec2man setup

Using the ec2man Python Module

Type python3 -m ec2man help for the following help message:

Usage: python3 -m ec2man help
       python3 -m ec2man setup
       python3 -m ec2man <Context> info
       python3 -m ec2man <Context> dshfile
       python3 -m ec2man <Context> <NodeID> <Operation> [Args]
       python3 -m ec2man <Context> all <Operation> [Args]

Operations:
    id: Get the instance ID string of the node
    prip:   Get the private ip address of the node
    pubip:  Get the public ip address of the node
    ssh:    Connect to a node through SSH; Can append with command
    put:    Use scp to put a file to the node's home directory
    rsync:  Use rsync to efficiently put a file to the node's home directory
    get:    Use scp to get a specific file from the node
    start:  Start the specified node
    stop:   Stop the specified node
    reboot: Restart the current node
    state:  Check the current state of the node

Tips:
    Before using this tool, create a 'profile' file 'ec2man/profile' which contains a line of your AWS ARN string and a line of profile name to use.
    Also create a 'machines' file 'ec2man/machines' which contains all the infos needed for setting up your instances.

These are frequently used functionalities:

Local$ python3 -m ec2man <Context> info                 # Get info about mahines under <Context>.
Local$ python3 -m ec2man <Context> <NodeIdx> ssh        # SSH connect to the node.
Local$ ...                                              # And a lot more...

Configure Password-less SSH Among Instances

With the help of ec2man module, you can configure password-less SSH connections among all instances in a convenient way:

Local$ python3 -m ec2man <Context> all put <PathToPrivateKey>
Local$ python3 -m ec2man <Context> all ssh "mv ~/<PrivateKeyFileName> ~/.ssh/id_rsa"

Previous page: Home | Next page: 2. Build Dorylus | Home: Home

Clone this wiki locally