Skip to content

Commit

Permalink
chore: update release_process.md
Browse files Browse the repository at this point in the history
  • Loading branch information
mansaj authored Sep 20, 2024
1 parent 1fa7975 commit 448c5d5
Showing 1 changed file with 18 additions and 19 deletions.
37 changes: 18 additions & 19 deletions docs/release_process.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ are merged back to `main`:
2. Make changes
3. Create a pull request to `main`
4. Address review comments
5. Merge pull request
5. Merge the pull request

```mermaid
%%{init: { 'theme': 'base', 'gitGraph': {'rotateCommitLabel': true} } }%%
Expand Down Expand Up @@ -57,11 +57,11 @@ merge back to `main` when they are ready.

## Release Timeline

The standard release interval for Monitor is 1 week, meaning every week there should be 1+ new version of the Monitor web app on the [Production][prod] environment. However, since we've started doing 1-click deploy to production, our release cycle can become more and more frequent.
The standard release interval for Monitor is one week, meaning there should be at least one new version of the Monitor web app on the [Production][prod] environment. However, since we've started doing 1-click deploy to production, our release cycle can become more and more frequent.

## Preview Deployment

Every time a PR is open, a docker image is created and deployed to the
Every time a PR is opened, a docker image is created and deployed to the
preview deployment environment powered by GCP Cloud Run / CloudSQL.
A brand new database is created and schema is migrated specific to that PR.
A brand new Cloud Run service is set up and cleaned up along with the
Expand All @@ -74,54 +74,54 @@ set up and changes are ready to be reviewed.
Every commit to `main` is automatically deployed to the [Stage][stage] server via Github Actions and Jenkins.

### PR Merges
PRs can only be merged when it passes all the required checks:
PRs can only be merged when they passes all the required checks:
* Lint
* Build
* Unit Tests
* E2E Tests
* Deploy Previews

Then a PR need to at least have 1 approval from ENGR team member in order to be merged into main.
A PR also needs at least one approval from an ENGR team member to be merged into `main`.

Once a PR is successfully merged:
1. ensure that the merge commit in main branch passes all the checks and a docker image is successfully deployed.
1. ensure that the merge commit in `main` branch passes all checks and a docker image is successfully deployed.
2. Jenkins will kick off the deployment of the latest built docker image to stage environment
3. A webhook will send status messages into the `#fx-monitor-engineering` channel.
* Watch for messages: `FX MONITOR STAGE STARTED` and `FX MONITOR STAGE COMPLETE`

## Release to Production
Before releasing to production, we need to assess the current state of our work on stage. We need to cross reference what's already on stage and what's greenlighted by QAs. To do this, we need to find the difference between what we released last time in production and what we currently have on stage.
Before releasing to production, we need to assess the current state of our work on stage. We need to cross-reference what's already on stage and what's been greenlit by QA. To do this, we need to find the difference between what was released last time in production and what we currently have on stage.

### Check the diff in Release Notes and notify the team
[make a release on GitHub][github-new-release] in order to check the difference.
1. Choose the tag you would want your relase be. Make this today's date (e.g., `2024.09.01`)
1. Choose the tag you want for your release. Use today's date (e.g., `2024.09.01`)
2. Type the same tag name for the release title (e.g., `2024.09.01`)
3. Click the "Generate release notes" button!
4. *DO NOT* press "Publish release" yet
5. Copy and Paste the release notes in the engineering slack channel so the team is aware
6. Go through the PRs, cross reference the tickets in the PRs with [jira][jira] board to see if QAs have OK'd the tickets. If anything's unclear, make sure to tag the author of the PR.
6. Go through the PRs, cross-reference the tickets in the PRs with the [Jira][jira] board to see if QA has approved the tickets. If anything is unclear, make sure to tag the author of the PR.
7. If anything has not been properly tested, make a note, and again, double check with the person
8. If everything looks good, proceed to release, otherwise refer to the section `Stage-fixes` below.

### Update Production Environment Variables
In the cases where we need to update or add new environment variables, we need to get help from SRE:
1. File an [SRE ticket][sre-board] for the env var change.
- In the title, make sure to mention "Production"
- Make sure to include the value and the right variable name
- Make sure to include the value and the correct variable name
- Make sure to specify if it's a `secret` or a regular variable
2. When appropriate, wait for SRE to make the changes first before doing a production release
2. When appropriate, wait for SRE to make the changes before proceeding with the production release.

### 1-click Production Release
After you push the tag to GitHub, you should also
[make a release on GitHub][github-new-release] for the tag.

1. After all the checks above look good, click "Publish release"
2. Go to the `main` branch and make sure all the checks succeeded
3. Go to [dockerhub][dockerhub] to make sure that a tag with today's date is present
3. Go to [DockerHub][dockerhub] to ensure that a tag with today's date is present.
4. Run [E2E cron][e2e] against stage (with the latest update)
* if there are errors, make sure the cause is understood
* fix the e2e errors or change the tests when appropriate before proceeding
6. Check stage Sentry and GCP error logs
6. Check the stage Sentry and GCP error logs
7. Run [1-Click Deploy Github Action][1-click deploy]
* Click on `Run workflow`
* `Branch:main` is selected
Expand All @@ -137,15 +137,14 @@ After you push the tag to GitHub, you should also

### Update Jira

On our [Jira][jira] board, take a look at the tickets listed under "Merged to main". If those were included in the release you just created, drag those tickets to either "Promoted to Prod" or "Done" column. This will notify QA that they can verify the behaviour on Prod if necessary.

If you're unsure whether a ticket was included in the release, ask the person it is assigned to to move it if needed.
On our [Jira][jira] board, review the tickets listed under "Merged to main." If those were included in the release you just created, drag those tickets to either the "Promoted to Prod" or "Done" column. This will notify QA that they can verify the behavior on Prod if necessary.

If you're unsure whether a ticket was included in the release, ask the assigned person to move it if needed.
## Stage-fixes

Ideally, every change can ride the regular weekly release "trains". But sometimes, not everything in `main` can go out. Since we've adopted feature flags, these scenarios are becoming rarer. However, we still cannot guarantee that they never happen.

Wherever feature flags aren't applicable, there are generally 2 scenarios we need to consider:
Wherever feature flags aren't applicable, there are generally two scenarios we need to consider:
1. If the diff in changes is minimal (eg. can be traced back to a PR or two), the easiest way is to revert
2. If the diff is not minimal, or a significant portion of the tickets haven't been QA'd:
* we can choose to delay the release (ask the team for consensus)
Expand All @@ -159,7 +158,7 @@ Wherever feature flags aren't applicable, there are generally 2 scenarios we nee

### Separate release branch
1. Create a branch on top of `main`
2. Work on taking out the features that should not be included (not feature flag)
2. Work on taking out the features that should not be included (not feature-flagged)
3. Create a Github [Release][github-new-release]
* In the release, make sure to pick your branch (`main` is default)
* Generate the release note, double check and make sure that it makes sense
Expand All @@ -170,7 +169,7 @@ Wherever feature flags aren't applicable, there are generally 2 scenarios we nee

After adding 1-click production deploy capability and broadly adopting [feature flags][feature-flags], we are looking into ways to increase our production release frequency. The main challenge here is to coordiate our QA effort with our latest stage CICD deployments.

We are starting to look into creating daily Github pre-releases via GHA, and once QA'd, have these deployed automatically / manually by base load engineers
We are starting to look into creating daily GitHub pre-releases via GHA, and once QA'd, having these deployed automatically or manually by base load engineers.

[prod]: https://monitor.firefox.com/
[stage]: https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/
Expand Down

0 comments on commit 448c5d5

Please sign in to comment.