EC2 usage

Setup IncQuery-D cluster on AWS

• Edit the security group of the instances to enable all incoming connections

• SSH with tunnel

• Switch to root: sudo -i

• Launch weave

  • In case of RM EC2 instance: weave launch
  • In case of all other EC2 instances: weave launch <FIRST EC2 INSTANCE IP>

• Start main node (RM):./start_node.sh rm [-use_ssh]

• Start worker nodes (NMs): ./start_node.sh <NODE_ID> [-use_ssh]

  • NODE_ID is a sequence from 1 to N

• Copy runtime to HDFS: docker exec $YARN_RM /usr/local/hadoop/copy_runtime_to_hdfs.sh

Forward ports to client

To reach cluster applications from your local machine, use ssh tunneling when connecting to EC2 instance.

Example:

• Hadoop (YARN and HDFS) web consoles: ssh -i <your key> -L 8088:127.0.0.1:8088 -L 50070:127.0.0.1:50070 ubuntu@incqueryd.incquerylabs.com