Apigee OPDK ansible (Under construction)

Centralize Edge operation tasks using one inventory file as reference for all tasks.

This project use as reference the following projects:

• Apigee opdk playbook setup ansible
• etp.

Table of contents

1. Requirements
2. Getting ready
   1. Configuration file
   2. Topology definition
   3. Generate Anisble inventory
   4. Generate response files
3. Apigee Edge Ops Tasks
   1. Install prereqs and the apigee-setup utility
   2. Port checking report
   3. Install Edge
   4. Update Edge
   5. Edge onboarding
   6. Create keystore in Edge and upload keystore certs
   7. Create a VirtualHost
   8. Run apigee-service commands accross planet
   9. Create Edge custom role
   10. Create user
   11. Fetch planet logs
   12. Run planet scan
   13. Ansible Ad-Hoc commands
4. Author

Requirements

Clone this repo:

# Move to your working path
git clone https://github.com/maurogonzalez/apigee-opdk-ansible.git
cd apigee-opdk-ansible.git

• ansible >= 2.2
• etp (to create topology diagram)

Go to the Installation wiki.

Getting ready

Configuration file

Under project root there's an env.yml, fill the required variables.

Property	Example	Description
license_path*	/home/gump/license.txt	License file absolute path including license file name.
apigee_user*	bubbagump	software.apigee.com user.
apigee_pwd*	mygumppassword	software.apigee.com password.
ssh_user	mgump	ssh user.
ssh_key	/home/gump/.ssh/gumptron.key	ssh key absolute path.
ssh_pwd	gumpsecret	ssh password. If not needed, leave the original value.
ssh_bastion_host	35.0.0.3	ssh bastion/jumpbox host. If not needed, leave the original value.
ssh_bastion_user	runforest	ssh bastion/jumbox user. If not needed, leave the original value.
ssh_bastion_key	/home/gump/runforest.key	ssh bastion/jumpbox key absolute path. If not needed, leave the original value.
pg_ram_in_mb*	4096	Postgresql machine memory required to set memory adjustments.
edge_version*	4.17.09	Edge version. At this point is 4.17.09.
opdk_admin_email**	forest@gump.com	Edge installation admin email.
opdk_admin_password**	forestpwd	Edge installation admin password.
opdk_smtp_mail_from**	forest@smtp.com	SMTP from.
opdk_smtp_skip**	n	"y" to skip SMTP configuration or "n".
opdk_smtp_user**	forest@user.com	SMTP user.
opdk_smtp_password**	forestpwd	SMTP password.
opdk_smtp_host**	smtp.gump.com	SMTP host.
opdk_smtp_port**	25	SMTP port.
opdk_smtp_ssl**	n	"y" or "n".
onboard_org_name***	BubbaGump	organization name for onboarding. If not needed, leave the original value.
onboard_admin_username***	forest@gump.com	org admin username (email format) for onboarding. If not needed, leave the original value.
onboard_admin_name***	Forest	org admin name for onboarding. If not needed, leave the original value.
onboard_admin_lastname***	Gump	org admin lastname for onboarding. If not needed, leave the original value.
onboard_admin_pwd***	forestpwd	org admin password for onboarding. If not needed, leave the original value.
onboard_env***	test	environment name for onboarding. If not needed, leave the original value.
onboard_vhost_alias***	api.forest.com	virtualhost alias for onboarding. If not needed, leave the original value.

* Required.

** Required if opdk_smtp_skip=n.

*** Required for onboarding.

Note: If you want to encrypt sensitive data: vault wiki.

Topology definition

First create an etp edge topology definition json file (example examples/topology-1dc-5n.json). This is the only file that you need to create.

Generate Anisble inventory

Ansible inventory file based on apigee-opdk-inventory-file.

Create an inventory file used by all Ansible playbooks and operation tasks:

$ ansible-playbook -e "topology_src=PATH_TO_TOPOLOGY_FILE" inventory.yml

Optionally, you can create the topology diagram in the same run setting the diagram variable to any value*:

$ ansible-playbook -e "topology_src=PATH_TO_TOPOLOGY_FILE diagram=1" inventory.yml

Create only the diagram*:

$ ansible-playbook -e "topology_src=PATH_TO_TOPOLOGY_FILE" diagram.yml

Variable	Example	Description
topolgy_src*	examples/topology.json	Path to topology json file.
diagram	1	Set to some value if you want to generate the diagram in the inventory play.

* Required

Find the files under:

• inventory/inventory_PLANET.INI
• reports/topology-PLANET.svg

Notes:

• * Diagram tasks, require nodejs and etp (Installation wiki).
• Do not modify the inventory file.

Generate response files

Generate response file per region used by the installation, upgrade and other ops tasks. Also create the onboarding response file:

$ ansible-playbook -e -i inventory/INVENTORY_FILE response_files.yml

Find the response files under:

• reports/PLANET/response_files/response_PLANET_REGION.cfg

Apigee Edge Ops Tasks

Install prereqs and the apigee-setup utility

Install the prerequisites described here and some more useful tools.

Installs the following packages across the planet:

• wget
• curl
• telnet
• nc
• nmap
• java-1.8.0-openjdk-devel
• apigee-service
• apigee-setup
• apigee-provision in management nodes

Uploads the license file, the response and onboarding files.

Sets Cassandra, Message Processor and Postgresql memory settings.

$ ansible-playbook -i inventory/INVENTORY_FILE prerequisites.yml

Port checking report

Create port report for a whole planet. This will test ports between nodes and create two CSV files:

• Connectivity report grouped by edge component.
• Compact report grouped by host.

$ ansible-playbook -i inventory/INVENTORY_FILE port_report.yml

Find the report files under:

• reports/port_connectivity_report_PLANET.csv
• reports/port_compact_PLANET.csv

Note: The hosts require nmap, it is installed in the above prerequisites playbook.

Install Edge

Install Edge components in the planet:

$ ansible-playbook -i inventory/INVENTORY_FILE -e "cmd=setup" setup.yml

Update Edge

Update Edge components in the planet, only if current version >= 4.16.09.

Set the value of the target version in your env.yml:

# filename: env.yml
...
edge_version: 4.17.09
...

And run:

$ ansible-playbook -i inventory/INVENTORY_FILE update.yml

Edge onboarding

Onboarding: create organization, environment, org admin user and default virtual host.

$ ansible-playbook -i inventory/INVENTORY_FILE onboard.yml

Create keystore in Edge and upload keystore certs

Apigee Keystores and Truststores.

Create a jar Keystore from key/cert pair, an Apigee Keystore in Edge and upload the JAR to Edge:

$ ansible-playbook -i inventory/INVENTORY_FILE \
  -e "keyalias=KEY_ALIAS keystore=KEYSTORE_NAME \
  ks_cert=PATH_TO_CERT ks_key=PATH_TO_KEY \
  ks_org=EDGE_ORG ks_env=EDGE_ENV" \
  keystore.yml

Variable	Example	Description
keyalias*	my_key_alias	Key alias.
keystore*	keystore	Keystore name.
ks_cert*	tls/server.crt	Path to cert file.
ks_key*	tls/server.key	Path to key file.
ks_org*	BubbaGump	Edge Org where the Keystore is going to be created.
ks_env*	test	Edge Env where the Keystore is going to be created.

* Required.

Create a VirtualHost

Create virtual host in an existing Edge Org/Env. Optionally TLS could be enabled with an existing keystore.

$ ansible-playbook -i inventory/INVENTORY_FILE \
  -e "keyalias=KEY_ALIAS keystore=KEYSTORE_NAME \
  vhost_name=VHOST_NAME vhost_aliases=COMMA_SEPARATED_ALIASES \
  org=EDGE_ORG env=EDGE_ENV tls_enabled=true" \
  vhost_tls.yml

Variable	Example	Description
vhost_name*	tls/server.crt	Path to cert file.
vhost_aliases*	tls/server.key	Path to key file.
org*	BubbaGump	Edge Org where the Keystore is going to be created.
env*	test	Edge Env where the Keystore is going to be created.
tls_enabled*	true	Enable TLS.
keyalias**	my_key_alias	Existing key alias.
keystore**	keystore	Existing keystore name.

* Required.

** Required if tls_enabled is set to any value.

Run apigee-service commands accross planet

Run an apigee-service command in particular components or across the planet.

$ ansible-playbook -i inventory/INVENTORY_FILE -e "cmd=COMMAND component=COMPONENT" setup.yml

Values for cmd:

• status
• start
• wait_for_ready
• stop
• restart

Values for component:

• zk (Zookeeper)
• cs (Cassandra)
• ds (Zookeeper and Cassandra)
• ldap (OpenLDAP)
• ms (Management Server)
• msldap (Management Server and OpenLDAP)
• r (Router)
• mp (Message Processor)
• rmp (Router and Message Processor)
• qs (QPIDD and Qpid Server)
• pg (Postgreql and Postgres Server)
• all (All Edge components)

Create Edge custom role

Edge Roles.

Create custom role in an Edge Org.

$ ansible-playbook -i inventory/INVENTORY_FILE \
  -e "api_action=roles org=ORG role=ROLE_NAME \
  customrole_path=ROLE_PATH" adminapi.yml

Variable	Example	Description
api_action*	roles	This must be roles to create new roles.
org*	BubbaGump	Edge Org where the role is going to be created.
role*	newrol	Role name.
customrole_path*	examples/customrole.json	Path to role in JSON format.

* Required.

Each Edge Org has some built-in roles:

• Organization Administrator: Super user. Has full CRUD access to resources in the organization. In an Edge for Private Cloud installation, the most powerful role is the System administrator role, which also has access to system-level functions that the Organization Administrator doesn't.
• Operations Administrator: Deploys and tests APIs; has read-only access to other resources.
• Business User: Creates and manages API products, developers, developer apps, and companies; creates custom reports on API usage; has read-only access to other resources.
• User: Creates API proxies and tests them in the test environment; has read-only access to other resources.

Create user

Edge Users.

Create user and add to an existing role.

$ ansible-playbook -i inventory/INVENTORY_FILE \
  -e "api_action=users org=ORG role=ROLE_NAME \
  user_email=USER_EMAIL user_password=USER_PASSWORD \
  user_name=USER_NAME user_lastname=USER_LASTNAME" \
  adminapi.yml

Variable	Example	Description
api_action*	users	This must be users to create new user.
org*	BubbaGump	Edge Org where the role is going to be created.
role	User	Edge role.
user_email*	forest@gump.com	User email (used as username).
user_password*	mypassword	User password.
user_name*	Forest	User name.
user_lastname*	Gump	User last name.

* Required.

Fetch planet logs

Tar all the edge logs from each node.

$ ansible-playbook -i inventory/INVENTORY_FILE logs.yml

Run planet scan

Run check commands across the planet:

• Edge pods: List servers for each Pod.

  • Central pod.
  • Gateway pod.
  • Analytics pod.

• Zookeeper: Run Zookeeper status commands.

  • ruok.
  • stat.
  • Zookeeper tree.

• Cassandra: Run Cassandra nodetool commands.

  • ring.
  • status.
  • statusthrift

• Postgres: Check master/standby nodes.

• Management ports check (self status): Get info from each Edge component.

  • Management Server (8080).
  • Router (8081).
  • Message Processor (8082).
  • Qpid Server (8083).
  • Postgres Server (8084).

• Analytics groups: List the Analytics groups and servers.

• Fetch customer/application files across the planet.

• Memory and disk usage.

$ ansible-playbook -i inventory/INVENTORY_FILE planet_scan.yml

Find the scan files:

• reports/scan:

Ansible Ad-Hoc commands

Ansible Ad-Hoc commands

Example:

$ ansible -i inventory/INVENTORY_FILE HOST_GROUP -m shell -a '/opt/apigee/apigee-service/bin/apigee-all status'

Script for inventory group hosts: Run echo "Hello node!" command example:

$ ./service.sh -i inventory/INVENTORY_FILE -g HOST_GROUP -c "echo 'Hello node!'"

Where:

• -h: help
• -i: ansible inventory path. Required
• -c: command to run in the node.
• -g: ansible inventory group

Default values:

• If not -c option provided, the default command is: /opt/apigee/apigee-service/bin/apigee-all status
• If not -g option provided, the default group is: planet

$ ./service.sh -i inventory/INVENTORY_FILE -g HOST_GROUP -c "echo 'Hello node!'"

* /opt/apigee/apigee-service/bin/apigee-all start|restart are not recommended since the start order is important: Starting apigee components

Author

If you have any questions regarding this project contact:
Mauro González jmajma8@gmail.com