This is a fork of AWS RDS Service Broker written by Pivotal Software Inc.
We have changed all source code includes to point to this repository.
This is a Cloud Foundry Service Broker for Amazon Relational Database Service (RDS) supporting MariaDB, MySQL and PostgreSQL RDS Databases.
More details can be found at this Pivotal P.O.V Blog post.
Using the standard go install (you must have Go already installed in your local machine):
$ go install github.com/alphagov/paas-rds-broker
$ rds-broker -config=<path-to-your-config-file>
This broker can be deployed using the Bosh release built in this repository (make bosh_release). On the GOV.UK PaaS, the release is created in the paas-rds-broker pipeline of the CI env.
You can patch an existing bosh environment by first logging into a bosh director and running the following:
$ make bosh_scp
Refer to the Configuration instructions for details about configuring this broker.
This broker gets the AWS credentials from the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. It requires a user with some IAM & RDS permissions. Refer to the iam_policy.json file to check what actions the user must be allowed to perform.
Configure and deploy the broker using one of the above methods. Then:
- Check that your Cloud Foundry installation supports Service Broker API Version v2.6 or greater
- Register the broker within your Cloud Foundry installation;
- Make Services and Plans public;
- Depending on your Cloud Foundry settings, you migh also need to create/bind an Application Security Group to allow access to the RDS DB Instances.
Application Developers can start to consume the services using the standard CF CLI commands.
Depending on the broker configuration, Application Depevelopers can send arbitrary parameters on certain broker calls:
Provision calls support the following optional arbitrary parameters:
| Option | Type | Description |
|---|---|---|
backup_retention_period |
Integer | The number of days that Amazon RDS should retain automatic backups of the DB instance (between 0 and 35) (*) |
character_set_name |
String | For supported engines, indicates that the DB instance should be associated with the specified CharacterSet (*) |
dbname |
String | The name of the Database to be provisioned. If it does not exists, the broker will create it, otherwise, it will reuse the existing one. If this parameter is not set, the broker will use a random Database name |
preferred_backup_window |
String | The daily time range during which automated backups are created if automated backups are enabled (*) |
preferred_maintenance_window |
String | The weekly time range during which system maintenance can occur (*) |
enable_extensions |
[]String | The names of the extensions which should be enabled. Supported extensions are specified by the plan, and the supplied list is combined with the set of default extensions defined by the plan. If this parameter isn't provided, the plan's default extensions will be enabled. (**) |
(*) Refer to the Amazon Relational Database Service Documentation for more details about how to set these properties
(**) Postgres only
Update calls support the following optional arbitrary parameters:
| Option | Type | Description |
|---|---|---|
apply_at_maintenance_window |
Boolean | Specifies whether the modifications in this request and any pending modifications are asynchronously applied as soon as possible (default) or if they should be queued until the Preferred Maintenance Window setting for the DB instance (*) |
backup_retention_period |
Integer | The number of days that Amazon RDS should retain automatic backups of the DB instance (between 0 and 35) (*) |
preferred_backup_window |
String | The daily time range during which automated backups are created if automated backups are enabled (*) |
preferred_maintenance_window |
String | The weekly time range during which system maintenance can occur (*) |
preferred_maintenance_window |
String | The weekly time range during which system maintenance can occur (*) |
reboot |
Boolean | Reboot the instance immediately. Any other parameter or change in the updated would be ignored. See https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html for details. |
force_failover |
Boolean | For HA failover during reboot. Only valid when used with reboot and for HA plans. See https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html for detais. |
update_minor_version_to_latest |
Boolean | Attempts to update the database to the latest available minor version supported by RDS as per the rds:DescribeDBEngineVersions API |
enable_extensions |
[]String | The names of the extensions which should be enabled. Supported extensions are specified by the plan, and the supplied list is combined with the set of default extensions defined by the plan. (**) |
disable_extensions |
[]String | The names of the extensions which should be disabled. Supported extensions are specified by the plan, and default extensions cannot be disabled. (**) |
(*) Refer to the Amazon Relational Database Service Documentation for more details about how to set these properties
(**) Postgres only
Reboot is performed by passing the custom parameter { "reboot": true } in an update. Pass { "reboot": true, "force_failover": true } to force failover in a HA instance.
It will not update the instance, so the new plan or custom parameters would be ignored.
The broker runs a number of housekeeping tasks. These need to be enabled on exactly one instance in your deployment by setting run_housekeeping to true in the config file.
The housekeeping task will delete old RDS snapshots which were created by this broker. It will search for snapshots older than keep_snapshots_for_days and with the matching Broker Name tag (config: rds_config.broker_name).
There are two forms of tests for the broker, the unit tests and the integration tests. The unit tests are run automatically by travis, but because the integration tests actually use the AWS RDS API they must be run manually or by an agent with AWS credentials.
To run the tests of this broker, you need a Postgres and MySQL running locally, without SSL. The connection details can be configured using environment variables - see the sqlengine test suites for more information.
To run the unit tests:
make unitThis handles the starting of Docker containers running MySQL and Postgres. It will also re-run the SQL engine tests against Postgres 10.5.
The Docker containers are removed after a successful test run. If you need to remove them you can use:
make stop_dbsThese tests must be run from within an AWS environment as they will attempt to both use the EC2 IMDS and connect directly to the RDS instance to verify it. They will create and delete some supporting resources:
- DB Subnet Group for the RDS instances, based on all of the subnets in the VPC of the instance that the tests run on
- EC2 Security Group for the test to connect to RDS instances, based on the subnet of the instance that the tests run on
They need IAM permissions for the above, everything listed in iam_policy.json, and the ability to delete RDS snapshots.
You can run them with:
make integration
It's best to resist the temptation to raise the tests' parallelism too far as AWS will throttle too many RDS API interactions for an account.
The RDS Broker generates and uses two sets of credentials: master and binding credentials.
The master credentials are generated when a DB instance is created and are used to create a database connection whenever an admin level access of the database only used by RDS-broker is required, including binding (equivalent of creating a new DB user), unbinding (equivalent of deleting a DB user), routing DB metrics, changing user passwords and rotating master credentials. It consists of:
- masterUsername - A random alphanumeric field generated at the point of instance creation.
- masterPassword - An SHA256 hashed alphanumeric field, generated based on instance id and a secret.
Binding allows an app to get access to a DB with limited user credentials. Whenever an app is deployed and configured to bind to a PaaS DB, a new set of credentials is generated and used to create a DB user, binding an app to the DB service, allowing access to the DB. Binding credentials consist of:
- username - this is an SHA256 hashed alphanumeric field, generated based on binding id (username)
- password - a random alphanumeric field
- Note on usernameold - we have recently changed our hashing algorithm from MD5 to SHA256. This function is to support the legacy binding credentials that are still using MD5 as hashing algorithm. When dropping a user (DropUser), we generate username (generateUsername) with the new hashing algorithm (SHA256), if there is no match, we try to use the old hashing algorithm (generateUsernameOld)
It is noted that masterPassword and binding username are deterministically generated by hashing the binding ID and secrets. Hence, if we change the hashing algorithm, it will have effect on both the master credentials and binding and unbinding of the instances, and may cause downtime if it is not handled properly.
Certain Postgres extensions (such as pg_stat_statements) require shared preload libraries
to be enabled. In RDS, preload libraries are specified as a parameter in the parameter group, and are enabled for all
databases in that group.
The RDS broker will create new parameter groups to support new combinations of extensions as necessary, or otherwise add new databases to existing groups with the right extensions.
Enabling or disabling extensions like pg_stat_statements that require shared preload libraries via an update call will apply a new parameter group. As a result, it also requires "reboot": true to be specified.
In the spirit of free software, everyone is encouraged to help improve this project.
Here are some ways you can contribute:
- by using alpha, beta, and prerelease versions
- by reporting bugs
- by suggesting new features
- by writing or editing documentation
- by writing specifications
- by writing code (no patch is too small: fix typos, add comments, clean up inconsistent whitespace)
- by refactoring code
- by closing issues
- by reviewing patches
We use the GitHub issue tracker to track bugs and features. Before submitting a bug report or feature request, check to make sure it hasn't already been submitted. You can indicate support for an existing issue by voting it up. When submitting a bug report, please include a Gist that includes a stack trace and any details that may be necessary to reproduce the bug, including your Golang version and operating system. Ideally, a bug report should include a pull request with failing specs.
- Fork the project.
- Create a topic branch.
- Implement your feature or bug fix.
- Commit and push your changes.
- Submit a pull request.
Copyright (c) 2015 Pivotal Software Inc. See LICENSE for details.