Safe Contract Changes Demo

This repository contains a sample full stack application to illustrate:

1. the risks when changing contracts between independently deployed components, and;
2. patterns for doing so safely

Application Overview

The application is a simple illustration of a common scenario - capturing, storing and retrieving data.

In this case, we are capturing a name, storing it (which allocates a new UUID) and then retriving it by its UUID.

Architecture

The application is made of three components:

• Next.js Single Page App
• Java Backend App
  • Spring Boot REST API
  • Hibernate Persistence
  • Liquibase Schema Management
• CDK Infrastructure
  • Cloudfront -> S3 to host the SPA
  • ECS Fargate to host the Java Service
  • RDS Aurora Serverless v2 to host the database

Contracts

A contract is a formal definition of how two components should interact. With software contracts, we use the terms client and server in a generic way:

1. client refers to the component initiating the interaction
2. server refers to the component responding to the interaction

There are a number of contracts in the application between components that might be affected by the timing of the deployments of the client and server:

1. API to add a name (POST): a. the SPA is the client, the Java app is the server
2. Writing the name to the database: a. the Java app is the client, the DB is the server (executing queries provided by the Java app)
3. Reading the name from the database: a. the Java app is the client, the DB is the server (executing queries provided by the Java app)
4. API to get a name (GET): a. the SPA is the client, the Java app is the server

Demo

These demos illustrate different ways of introducing a new feature - instead of capturing a single name, we now need to capture a 'full name' as two separate fields: firstName and lastName.

The WRONG Way

To demonstrate the wrong way, first checkout the tag 1 commit which contains a baseline version of the application before the change and deploy it:

git checkout 1
cd infra
cdk deploy

(This assumes you have AWS credentials setup in your environment and a number of other common dependencies, including cdk installed globally and docker.)

Once deployed, the 'base' version of the application will be available to you in your AWS account.

Next, checkout the single commit change (tag 2) and start a test loop:

git checkout 2
cd tests/e2e
npm run test:aws:loop

This will start a playwright test in a loop that captures 10 names and then repeats.

NOTE: The Tests are written to operate like a human - they are flexible and will pass with either the 'single field' version or the 'two fields' versions of the app. A human will do their best to make use of the screen presented to them - so do these tests.

If you run the tests against the first version of the app, they will pass. And if you run them against the second version of the app, they will pass. The interesting thing is: what happens if you run them while the deployment is in mid-flight? This attempts to demonstrate the the experience of a user who lands on your application while the deployment is underway.

To find out, while the test loop is running, in a separate terminal, re-deploy using (from the repo root):

cd infra
cdk deploy

After a few minutes, the tests will failing.

If you re-start them immediately, they will still fail.

However, if you wait long enough, and re-start the tests, they will start passing.

The reason for the failure

There are a few possible reasons for the failures you've seen:

1. The SPA completed deployment before the server, and was passing down two fields and expecting two back, to an app that was still expecting 1 field and returning 2
2. The server had completed deployment and updated the database schema. The SPA had not yet completed deployment, and was randomly interacting with an old version of the server that hadn't been 'rolled over' yet, and the old version of the server was unable to make use of the new database schema.

Eventually, once all deployments are completed and the versions are aligned, the tests will pass again.

The SAFE Way

The SAFE way to deploy contract changes is to use the "Parallel Change" or "Expand/Contract" pattern. It is a simple three steps:

1. Expand: Change the Server so that it is backwards and forwards compatible, supporting clients of both versions
2. Change: Change the client so that it is consuming only the new version
3. Contract: Clean up the Server so that it only supports the new version

API Demo

This repository contains an illustration of this pattern as applied to the REST APIs.

If you ran the 'WRONG way' demo above, please reset your environment:

git checkout reset
# Reset the db
cd db
./reset_aws_db.sh
cd ../infra
cdk deploy

Once complete you will have the 'single field' version of the app.

The three steps are marked with tags:

• api-step-1
  • Update the server to accept both single name field and a fullName object with firstName and lastName fields
  • If the fullName field is supplied, construct a single name by appending the names with a space separator and store it on the single name field.
  • When returning the result, return both the name and fullName fields with the latter formed by splitting on whitespace to extract the first and last names.
• api-step-2
  • Update the app to only send and only accept the fullName field and no longer know anything about the name field
• api-step-3
  • Update the server to only accept and return the fullName field
  • There is still an adaptor function in both directions in the domain layer which accepts and returns the fullName objects, but stores the single name field.

To ensure each step's deployment is safe, you can:

1. Checkout the step by tag (replace ? with the step number you are up to)

   git checkout api-step-?
   

2. Start the test loop:

   cd tests/e2e
   npm run test:aws:loop
   

3. Deploy:

   cd infra
   cdk deploy
   

After the deployment is complete, you can stop the tests with CTRL-C.

DB Demo

This repository also contains an illustration of this pattern as applied to Database interactions via an ORM.

The DB change expects that you have completed the API change. Please do those steps first then return to this point.

The expand/contract pattern with ORMs can be a little confusing at first, because although we are following the 'step 1 - change the server to be forwards and backwards compatible' we are still changing some of the client in step 1. This is because the client generates code that executes on the server in the form of SQL scripts, so we need to make the necessary changes to alter the SQL scripts that get created and executed inside the DB.

The three DB change steps are marked with tags:

• db-step-1
  • Add the two new columns, ensuring they are nullable
  • Update the server Entity so that on writes it writes to all columns: old and new
  • Update the server Entity so that on reads it will try and use the new columns first, and fall back to the old columns if the new ones are not populated
  • This changes ensure that if a request is handled by a new instance, it will generate data that is still usable by an old instance. And if a request is handled by an old instance, even though it will leave the new fields as null, a new instance can still use that record.
• db-step-2
  • At this point, all nodes should be updated, so all new records should have the new fields populated, and there should be no more new records with unpopulated new fields.
  • Execute a script to copy data from old fields to new fields for all old records that don't yet have data in the new fields
  • Ensure the old fields are nullable
  • Remove awareness of the old fields from the server Entity. New nodes will only be reading and writing to the new fields. Old nodes will read and write to both old and new fields, but as all records now have populated new fields, they will, practically speaking, only be using the new fields.
• db-step-3
  • Now there should be no nodes that know anything about the old fields
  • It is safe to drop the old fields from the db

To ensure each step's deployment is safe, you can:

1. Checkout the step by tag (replace ? with the step number you are up to)

   git checkout db-step-?
   

2. Start the test loop:

   cd tests/e2e
   npm run test:aws:loop
   

3. Deploy:

   cd infra
   cdk deploy
   

After the deployment is complete, you can stop the tests with CTRL-C.