SCT AMI maintenance page

Contact: Amos Kong <amos@scylladb.com>

Background

SCT will install and configure test utils on clean Loader and Monitor instance by node_setup(). We want to skip the environment setup for Loader and Monitor. For loader, currently we want to solve a known c-s issue by installing shlomi's fixed c-s in Loader AMI. For AMI, it can help to save our test time.

AMI for Loader

• maintain instance: scylla-qa-loader-ami-latest (instance-id: i-0248929691a395ca8, us-east-1)
• related ticket: https://trello.com/c/QD8ZYt3v/308-creation-of-ami-loader
• prepare steps:
  • setup loader by a general SCT job, use it as base
  • remove install scylla-tools-java packages (scylla-tools-core, scylla-tools)
  • build https://github.com/slivne/scylla-tools-java/tree/slivne_fix_cs_userdefined_retry

git clone https://github.com/slivne/scylla-tools-java
cd scylla-tools-java
git checkout slivne_fix_cs_userdefined_retry
bash -x ./reloc/build_reloc.sh
bash -x ./reloc/build_rpm.sh

[amos@amos-centos7 scylla-tools-java]$ find build|grep rpm
build/redhat/scylla-package/dist/redhat/build_rpm.sh
git clone https://github.com/slivne/scylla-tools-java
cd scylla-tools-java
git checkout slivne_fix_cs_userdefined_retry
bash -x ./reloc/build_reloc.sh
bash -x ./reloc/build_rpm.sh

[amos@amos-centos7 scylla-tools-java]$ find build|grep rpm
build/redhat/scylla-package/dist/redhat/build_rpm.sh
build/redhat/RPMS/noarch/scylla-tools-666.development-20190116.c410e90.noarch.rpm
build/redhat/RPMS/noarch/scylla-tools-core-666.development-20190116.c410e90.noarch.rpm
build/redhat/SRPMS/scylla-tools-666.development-20190116.c410e90.src.rpm
build/redhat/BUILD/scylla-tools-666.development/dist/redhat/build_rpm.sh
build/redhat/RPMS/noarch/scylla-tools-core-666.development-20190116.c410e90.noarch.rpm
build/redhat/SRPMS/scylla-tools-666.development-20190116.c410e90.src.rpm
build/redhat/BUILD/scylla-tools-666.development/dist/redhat/build_rpm.sh

• install built scylla-tools-java packages to the builder

[amos@amos-centos7 scylla-tools-java]$ sudo yum install build/redhat/RPMS/noarch/scylla-tools-666.development-20190116.c410e90.noarch.rpm build/redhat/RPMS/noarch/scylla-tools-core-666.development-20190116.c410e90.noarch.rpm

• base
  • us-east-1: ami-09db4bf9d8a70042d (v0)
• v1 (Thu Jan 31 22:56:11 CST 2019):
  • us-east-1: ami-0e40245be9878b0b8 (v1)
  • eu-west-1: ami-0dfdd5cbd9df7f135 (v1)
• v2 (Tue Feb 12 16:08:36 CST 2019):
  • us-east-1:scylla-qa-loader-ami-v2 ami-07797ae1c88489b80
  • eu-west-1: ami-05a77f776bd5f9f4d
• v3 (Wed Mar 20 20:37:21 CST 2019):
  • us-east-1: scylla-qa-loader-ami-v3 ami-050c06c3e20036edc
  • eu-west-1: ami-0006222380fb72d8d
• v4 (Thr Apr 18 10:09:46 CST 2019):
  • us-east-1: scylla-qa-loader-ami-v4(used in sct yaml) ami-059dd02510b5f5841
  • us-west-1: scylla-qa-loader-ami-v4 ami-0dad569a4bdbc86a1
  • us-west-2: scylla-qa-loader-ami-v4 ami-0011d4855d80b3127
  • es-west-1: scylla-qa-loader-ami-v4(used in sct yaml) ami-07d0df7186ef5adc5
• v7 (Wed Dec 18 20:47:07 CST 2019): by Amos
  • us-east-1: 'ami-00410516d864c4a80' # Loader dedicated AMI
  • eu-west-1: 'ami-0a9a4dc9eb5fbf9e8' # Loader dedicated AMI
  • us-west-2: 'ami-0c6fe6ef0d47ae428' # Loader dedicated AMI

AMI for Monitor

• maintain instance: SCT-Monitor-AMI-latest (instance-id: i-0af4890b9993dcea7, eu-west-1)
• related ticket: https://trello.com/c/wPRhz8fr/495-vpc-subnets-builders-rearrange-all-work-only-with-private-ip-except-multidc
• prepare steps:
  • use a general SCT job to setup a monitor, use it as base
  • rm -rf /var/lib/scylla/scylla-grafana-monitor*
  • rm -rf /var/lib/scylla/branch-*
• base:
  • eu-west-1: ami-0876bfff890e17a06 (v0)
• v1:
  • us-east-1: ami-07af7fbdaf6768302 (v1)
  • eu-west-1: ami-0876bfff890e17a06 (v1)
• v2 (Thu Jan 31 22:56:11 CST 2019):
  • us-east-1: ami-01d6b0d7e0d98969b (v2)
  • eu-west-1: ami-0f478f18d613e3662 (v2)
• v3: (https://wiki.centos.org/Cloud/AWS) clean official AMI (with ENA support)
  • name: CentOS Linux 7 x86_64 HVM EBS ENA 1901_01
  • us-east-1: ami-02eac2c0129f6376b
  • us-west-1: ami-074e2d6769f445be5
  • us-west-2: ami-01ed306a12b7d1c96
  • eu-west-1: ami-0ff760d16d9497662
  • eu-west-2: ami-0eab3a90fc693af19

Reference:

• add new region eu-west-1 and use dedicated AMI for loader/monitor #779
• https://trello.com/c/wPRhz8fr/495-vpc-subnets-builders-rearrange-all-work-only-with-private-ip-except-multidc
• https://trello.com/c/QD8ZYt3v/308-creation-of-ami-loader