This repository defines the Open Targets Platform infrastructure, using Hashicorp Configuration Language (HCL) and Terraform.
As shown in the figure above, a deployment unit consists of two data backends, based on Elastic Search and Clickhouse services. Set to automatically scale up on demand and self-heal, as part of the managing activities in their scoping regional instance group.
These services are accessible by the backend API via their respective Internal Load Balancers, which not only distributes the requests among available instances, but also prevents a broken instance from receiving any traffic.
Open Targets platform API services are deployed as a collection of instances in a regional group, set behind a Global Load Balancer, that serves requests from the internet to these services via HTTP and HTTPS, being SSL terminated at this infrastructure entry point.
The platform web frontend is deployed in a Google Cloud Storage Bucket, not shown in the picture above, that is also behind this Global Load Balancer.
We use Google Cloud DNS services for all domain names deployed via this infrastructure definition.
Open Targets Platform Infrastructure has been defined using Terraform Modules.
The platform has been broken down into the following components:
Each component is defined and encapsulated as a submodule that can be reused independently, in any Google Cloud environment run, or not, by Open Targets.
These modules are accessible by the community via our Open Source license, as well as their underlying machine images (via their corresponding licenses).
This repository itself makes use of them as a Terraform Root Module, to build what constitutes Open Targets Platform. In addition, this root module is also available to the community via our Open Source license, thus, anyone can use it for deploying the platform in any Google Cloud project environment.
This also simplifies our development, testing, QA and staging workflows internally at Open Targets, by doing a lot of heavy lifting for our backend and frontend engineers.
Prior to working with this infrastructure definition, you need to make sure that you have:
- Terraform v1.3.9 or later
- Google Cloud SDK 420.0.0 or later, with valid credentials
- Makefile enabled environment, with common command line tools, e.g. wget
Clone the repository, if you have GitHub Cli:
gh repo clone opentargets/terraform-google-opentargets-platform
To interact with GCP services it is necessary to be authenticated. Follow the directions in the google module documentation.
If you are already signed in to the gcloud utility (check with gcloud auth list) executing the following command should be sufficient:
gcloud auth application-default login
Terraform uses a state file that allows it to track the managed infrastructure.
The information is stored in a file, and this file can reside in the local file system, or somewhere else, e.g. a Google Cloud Bucket.
This is known as Terraform backend, and, by default, it is set to use a location in one of our buckets.
When you clone this repository, and want to use this code outside Open Targets, you need to update the file backend.tf with the details of where you'd like to save your Terraform state.
To put together all the elements that make up an Open Targets Platform deployment for its different versions, either production releases or development branches, we use the idea of Deployment Contexts, defined in the profiles folder, and used to define the different versions of the platform that we want to deploy.
In a nutshell, a Deployment Context is a set of variables that define the version of the platform to be deployed, and the infrastructure that will be used to host it, i.e. all the configuration parameters that can be customized for deploying the platform.
To simplify the process of working with these Deployment Contexts, we have split its contents into different layers.
The first layer is the one that defines the underlying cloud provider, in this case, Google Cloud, and it is implemented in the file 01-defaults-gcp.auto.tfvars.
This file contains the default values for all the variables that are specific to Google Cloud, and are used by the modules that make up the platform.
The second layer defines the geometry of the machines that will be used to host the platform, and it is implemented through a series of files (each related to a specific platform service or component):
10-vm_geometry-api.auto.tfvars, for the API.10-vm_geometry-clickhouse.auto.tfvars, for the Clickhouse data backend.10-vm_geometry-elastic_search.auto.tfvars, for the Elastic Search data backend.10-vm_geometry-web_server.auto.tfvars, for the machines that host the web frontend.
As an example of machine geometry settings, the following snippet shows the values for the API machines:
// API Node --- //
config_vm_api_vcpus = "2"
config_vm_api_mem = "7680"
config_vm_api_boot_disk_size = "10GB"The third layer defines the machine's persona that will be used in the deployment, and it is implemented through a series of files (each related to a specific platform service or component):
11-vm_persona-api.auto.tfvars, for the API.11-vm_persona-clickhouse.auto.tfvars, for the Clickhouse data backend.11-vm_persona-elastic_search.auto.tfvars, for the Elastic Search data backend.11-vm_persona-web_server.auto.tfvars, for the machines that host the web frontend.
As an example of machine persona settings, the following snippet shows the values for the API machines:
// API Node --- //
// By default, we use the development configuration, where the provisioning model for VMs is preemptible.
config_vm_api_flag_preemptible = trueAcross the different layers, the configuration parameters corresponding to a development environment are the ones used by default.
The fourth layer defines the operating system that will be used in the deployed machines, and it is implemented through a series of files (each related to a specific platform service or component):
20-vm_os-api.auto.tfvars, for the API.20-vm_os-clickhouse.auto.tfvars, for the Clickhouse data backend.20-vm_os-elastic_search.auto.tfvars, for the Elastic Search data backend.20-vm_os-web_server.auto.tfvars, for the machines that host the web frontend.
As an example of operating system settings, the following snippet shows the values for the API machines:
// API node --- //
config_vm_api_image = "cos-stable"
config_vm_api_image_project = "cos-cloud"The fifth layer defines the software that we'll run on the deployed machines, on top of the OS, and it is implemented through a series of files (each related to a specific platform service or component):
30-vm_sw-api.auto.tfvars, for the API.30-vm_sw-clickhouse.auto.tfvars, for the Clickhouse data backend.30-vm_sw-elastic_search.auto.tfvars, for the Elastic Search data backend.30-vm_sw-web_server.auto.tfvars, for the machines that host the web frontend.
As an example of software settings, the following snippet shows the values for the Elastic Search machines:
// Elastic Search node --- //
// This is the Elastic Search Docker image we will use for the Elastic Search node.
config_vm_elastic_search_version = "7.10.2"The sixth layer defines the security policies and related configuration for the deployment of the infrastructure.
It is implemented in the file 50-security.auto.tfvars.
In the current infrastructure iteration, two areas are covered:
- The access to API
- The access to the web frontend
The seventh layer defines the DNS configuration for the deployment, in file 60-dns.auto.tfvars.
The following snippet shows the default DNS configuration, which is used for the development environment:
// By default, we use the development DNS configuration
config_dns_project_id = "open-targets-eu-dev"
config_dns_subdomain_prefix = "dev"
config_dns_managed_zone_name = "opentargets-xyz"
config_dns_managed_zone_dns_name = "opentargets.xyz."
config_dns_platform_api_subdomain = "api"This settings are related to the Cloud DNS service configured in open-targets-eu-dev project, and the opentargets-xyz managed zone, building the following DNS URL: api.dev.opentargets.xyz.
The eighth layer defines the configuration for the Global Load Balancer that will be used to expose the platform to the outside world, in file 70-glb.auto.tfvars.
The main aspect of the GLB that can be customized is related to whether we want to use a CDN or not, which is controlled by the following parameter:
// By default, we use the development configuration
config_glb_webapp_enable_cdn = falseThe ninth layer defines the sitemaps for the web frontend, in file 80-sitemaps.auto.tfvars.
The following snippet shows the default sitemaps configuration, which is used for the development environment:
// Sitemaps default configuration, it matches Development Environment
config_webapp_sitemaps_repo_name = "opentargets/ot-sitemap-cli"
config_webapp_sitemaps_release = "1.1.0"
config_webapp_sitemaps_bigquery_table = "platform_dev"
config_webapp_sitemaps_bigquery_project = "open-targets-eu-dev"This setting uses a tool at GitHub repository opentargets/ot-sitemap-cli to generate the sitemaps for the web frontend, based on the data of the BigQuery dataset platform_dev in open-targets-eu-dev project.
The tenth layer defines the environment variables for the web frontend, in file 81-web_env.auto.tfvars.
The following snippet shows the default web frontend environment configuration, which is used for the development environment:
// Default configuration for the web application is the Development Environment
config_webapp_deployment_context_map = {
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_URL_API = "'https://api.platform.dev.opentargets.xyz/api/v4/graphql'"
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_URL_API_BETA = "'https://api.platform.dev.opentargets.xyz/api/v4/graphql'"
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_EFO_URL = "'/data/ontology/efo_json/diseases_efo.jsonl'"
}
// Robots.txt profile --- //
config_webapp_robots_profile = "default"
// Web Application Customisation Profile --- //
// Using the default profile
// config_webapp_custom_profile = "default.js"It defines environment configuration that will be shipped with the frontend to the web servers, as well as which robots.txt profile and custommization profiles will be used.
The eleventh layer defines the configuration for the monitoring of the platform, in file 90-monitoring.auto.tfvars.
This layer sets, among other things, development and monitoring facilities for the deployed infrastructure, such as:
- Enable/disable SSH access to the machines.
- Enable/disable additional VMs for inspection and monitoring purposes from within each deployment region.
The following snippet shows the default monitoring configuration, which is used for the development environment:
// Development facilities //
// Development mode will enable the following features:
// - SSH access to deployed instances
config_set_dev_mode_on = true
// Inspection will enable the following features:
// - An SSH enabled instance within the same VPC as the deployed instances, for every deployed region
//config_enable_inspection = trueThis is the final layer that defines the deployment context, which groups together those elements that change in-between releases (mainly).
It is implemented through a configuration management system based on the use of profiles, which are defined in the profiles folder, and defined as deployment_context.<profile_name>.
For example, the profiles/deployment_context.2302 profile defines the deployment context for Open Targets Platform 23.02 release, as it can be seen in the following snippet:
// --- PRODUCTION Open Targets Platform ---//
// --- Release Specific Information (THIS IS THE MAIN PLACE WHERE THINGS CHANGE BETWEEN PRODUCTION RELEASES) --- //
// Regions
config_deployment_regions = [ "europe-west1" ]
// Elastic Search configuration
config_vm_elastic_search_image = "mbdevplatform-230214-031922-es"
// Clickhouse configuration
config_vm_clickhouse_image = "mbdevplatform-230214-031922-ch"
// API configuration
config_vm_platform_api_image_version = "dev23.02.7"
config_vm_version_major = "23"
config_vm_version_minor = "02"
config_vm_version_patch = "7"
config_vm_data_year = "23"
config_vm_data_month = "02"
config_vm_data_iteration = "0"
// Web App configuration
config_webapp_release = "v0.3.4"
// Web App Data Context
config_webapp_bucket_name_data_assets = "open-targets-data-releases"
config_webapp_data_context_release = "23.02"
// -[END]- Release Specific Information --- //
// --- This section is common to all production releases, unless custom changes need to be made --- //
// Deployment configuration
config_release_name = "production"
config_project_id = "open-targets-platform"
// DNS
config_dns_project_id = "open-targets-prod"
config_dns_subdomain_prefix = null
config_dns_managed_zone_name = "opentargets-org"
config_dns_managed_zone_dns_name = "opentargets.org."
config_dns_platform_api_subdomain = "api"
// Web App configuration
config_webapp_deployment_context_map = {
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_URL_API = "'https://api.platform.opentargets.org/api/v4/graphql'"
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_URL_API_BETA = "'https://api.platform.opentargets.org/api/v4/graphql'"
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_EFO_URL = "'/data/ontology/efo_json/diseases_efo.jsonl'"
DEVOPS_CONTEXT_PLATFORM_APP_CONFIG_GOOGLE_TAG_MANAGER_ID = "'GTM-WPXWRDV'"
}
// Robots.txt profile
config_webapp_robots_profile = "production"
// Sitemaps generation
config_webapp_sitemaps_bigquery_table = "platform"
config_webapp_sitemaps_bigquery_project = "open-targets-prod"
// Machines' Persona configuration
// For Production, we use 'on-demand' provisioning model
config_vm_api_flag_preemptible = false
config_vm_clickhouse_flag_preemptible = false
config_vm_elasticsearch_flag_preemptible = false
config_vm_webserver_flag_preemptible = falseThe Release Specific Information section is the one that changes in-between releases, and it is the one that should be modified when a new release is deployed. It contains information about the regions where the infrastructure will be deployed, the images that will be used for the data backend VMs, the version of the API and Web App that will be deployed, and the data context that will be used.
To simplify operations related to working with this infrastructure definition, we have implemented a Makefile helper with the following operations:
Usage:
make
help show help message
status Show the current status of the deployment context
set_profile Set the profile to be used for all the operations in the session (use parameter 'profile')
update_linked_profile Update a linked deployment context profile to point to a new one, e.g. 'production-platform' -> '23.02', (use parameters 'profile' and 'link_to_profile')
clone_profile Clone an existing profile to a new one, starting with an empty workspace (state), and activate the new deployment context profile, as well as its corresponding workspace, use parameters 'profile' and 'new_profile'
delete_profile Delete an existing profile, use parameter 'profile'
unset_profile Unset the currently active profile
clean_backend Clean Terraform Backend Cache
clean Clean up all the artifacts created by this helper (profile, backend, etc.)In this example, we are going to show the steps to work with the development deployment context, dev-platform.
The first step after clonning the repository, is to set the active profile
make set_profile profile=dev-platformThis command will set the active profile to dev-platform, and it will also activate the corresponding Terraform Environment (Workspace).
A new file, deployment_context.auto.tfvars will be created as a symbolic link to the corresponding profile file, profiles/deployment_context.dev-platform.
Now we initialize the Terrafom backend for the active profile
terraform initTo plan / apply changes to the infrastructure, based on the active profile, we use standard Terraform commands:
terraform plan
# To apply changes
terraform applyOnce we are done with the changes, we can deactivate the active profile by running the following command:
make unset_profileThis will prevent any accidental changes to the infrastructure.
To create a new deployment context profile, based on an existing one, we can use the clone_profile target, e.g. for a new profile named test based on the dev-platform profile, we can run the following command:
make clone_profile profile=dev-platform new_profile=testThis will create a new profile file, profiles/deployment_context.test, activate the profile, and create a new Terraform Environment (Workspace) for it (with the same name as the profile).
A linked profile is a profile that points to another profile, and it is used to simplify the process of updating the infrastructure when new releases are deployed.
For example, the production-platform profile is a linked profile that points to the 2302 profile, which is the profile that corresponds to the latest release of the Open Targets Platform (at the time of writing this document).
production-platform linked profile is, actually, the one we use for our production deployment.
To update the production-platform profile to point to a new profile, we can use the update_linked_profile target, e.g. to update the production-platform profile to point to a new profile with name 2304 (which corresponds to the coming release 23.04) we can run the following command:
make update_linked_profile profile=production-platform link_to_profile=2304Deleting an existing profile will:
- Delete the corresponding Terraform Environment (Workspace) (with automatic destruction of all the resources tracked by Terraform in that environment)
- Delete the corresponding profile file
As an example, to delete test profile, we can run the following command:
make delete_profile profile=testTwo targets are provided for housekeeping related tasks:
clean_backend: This target will clean the Terraform backend cache, by removing all the files in the.terraformdirectory.clean: This target will clean all the artifacts created by this helper, including the active profile (deactivate / unset) and the Terraform backend cache.
This section explains the different sections of the deployment context file, and the parameters that can be used to configure the deployment, through the different definition layers.
config_gcp_default_region, default deployment region when no region is specified, this parameters is related to the cloud provider configuration, in this case Google Cloud Platform.
config_gcp_default_zone, default deployment zone when no zone is specified, this parameters is related to the cloud provider configuration, in this case Google Cloud Platform.
config_project_id, Google Cloud Project ID that will be the destination for deployed resources.
API nodes used as example
config_vm_api_vcpus, CPU count to be allocated for every Open Targets Platform API instance.
config_vm_api_mem, amount of memory to be allocated for every Open Targets Platform API instance (MiB).
config_vm_api_boot_disk_size, VM boot disk size to attach to the Open Targets Platform API instances.
API nodes used as example
config_vm_api_flag_preemptible, flag to indicate if the Open Targets Platform API instances should use SPOT or ON_DEMAND provisioning models.
config_vm_api_image, VM image ID for Open Targets Platform API instances.
config_vm_api_image_project, VM images may be hosted in a project different than the recipient of the resources being deployed, which ID is specified here.
Elastic Search nodes used as example
config_vm_elastic_search_version, Docker image version for running Elastic Search within the VM isntances.
config_security_api_enable, whether to restrict access to the deployed API based on the provided CIDR listing or not.
config_security_webapp_enable, whether to restrict access to the deployed web application based on the provided CIDR listing or not.
config_dns_project_id, ID of the Google Cloud Project that hosts the Cloud DNS services where DNS entries will be registered.
config_dns_managed_zone_name, name of the Cloud DNS managed zone where entries can be registered.
config_dns_subdomain_prefix, if supplied, this parameters allows for the resources be "scoped" within a subdomain in the DNS, this way, multiple deployments can share the main root domain name.
config_dns_managed_zone_dns_name, root domain name associated to the managed DNS zone.
config_dns_platform_api_subdomain, which subdomain to use fot Open Targets Platform API services, default is 'platform'.
config_glb_webapp_enable_cdn, whether or not a CDN should be used for serving the web application.
config_webapp_sitemaps_repo_name, GitHub repo where to find the sitemaps generator, e.g. opentargets/ot-sitemap-cli
config_webapp_sitemaps_release, GitHub release number to use for sitemaps generation.
config_webapp_sitemaps_bigquery_table, BigQuery table where the data for generating sitemaps can be found.
config_webapp_sitemaps_bigquery_project, project where BigQuery table for sitemaps is set.
config_webapp_deployment_context_map, this is a simple key-value collection that will be injected as configuration in the web application when deployed.
config_webapp_robots_profile, robots.txt profile that should be set in the given deployment.
config_webapp_custom_profile, web application customisation profile to be set in the given deployment.
config_set_dev_mode_on, when development mode is active, SSH traffic to all deployed VMs is enabled, and inspection VMs are deployed.
config_enable_inspection, when true, an extra VM per region will be deployed, for internal access to deployed infrastructure.
Values present in this layer, overwrite values from the underlying layers.
This listing is not exhaustive, but it includes the most important parameters, not mentioned earlier, that can be used to configure the deployment.
config_release_name, its value will be used for prefix-scoping names / IDs of deployed resources
config_deployment_regions, a list of Google Cloud regions for deploying Open Targets Platform across.
config_vm_elastic_search_image, VM image ID for Open Targets Elastic Search instances.
config_vm_clickhouse_image, VM image ID for Open Targets Clickhouse instances.
config_vm_platform_api_image_version, Open Targets Platform API image version to deploy.
config_webapp_release, GitHub tag release.
config_webapp_repo_name, GitHub repository where to fetch the bundle from.
config_webapp_bucket_name_data_assets, bucket name where to find the data assets that will be used for the web application data context.
config_webapp_data_context_release, subfolder within the data bucket where to find that web application data context.
config_webapp_webserver_docker_image_version, Docker image version for running the web server that attends web application requests.
(TODO - Security configuration parameters)
Once the deployment process has been successfully completed, the following details about the resources are revealed.
network_region_subnet_mapping, a custom VPC is created for every deployment context of Open Targets Platform, and, as a result, information on the created custom subnets is output here.
elastic_search_deployments, information on the Elastic Search deployments, mainly their Google Cloud internal load balancers IP addresses, see also submodule documentation.
clickhouse_deployments, information on the Clickhouse deployments, mainly their Google Cloud internal load balancers IP addresses, see also submodule documentation
api_deployments, output information from the API submodule is forwarded here, see also submodule documentation
webapp_deployment, output information from the webapp submodule is forwarded here, see also submodule documentation
debug_glb_platform, details on the deployed platform GLB are provided through this output parameter.
dns_records, provides the details about the created DNS records.
inspection_vms, this output parameter offers information about the inspection VMs deployed within the infrastructure.
Infrastructure visual diagrams use AWS icons and visual elements, but their meaning in Open Targets Google Cloud Infrastructure is the same, from the conceptual point of view.
Copyright 2014-2024 EMBL - European Bioinformatics Institute, Genentech, GSK, MSD, Pfizer, Sanofi and Wellcome Sanger Institute
This software was developed as part of the Open Targets project. For more information please see: http://www.opentargets.org
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.