Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GHA: Build and Publish to Public ECR and Build Nix #49

Merged
merged 16 commits into from
Oct 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/actions/setup-ecr-buildx/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Setup ECR / Buildx
description: Composite action to setup requirements for ECR and buildx for docker pushes

inputs:
registry-type:
description: ECR Registry Type
default: private
iam-role-to-assume:
description: IAM Role to Assume
required: true

outputs:
ecr_registry:
description: ECR Registry
value: ${{ steps.login-ecr.outputs.registry }}

runs:
using: "composite"
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ inputs.iam-role-to-assume }}
role-session-name: github-action-session
aws-region: us-east-1

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
with:
registry-type: ${{ inputs.registry-type }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@ecf95283f03858871ff00b787d79c419715afc34 # v2.7.0
68 changes: 68 additions & 0 deletions .github/workflows/build-and-push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Build and Push

# Builds and pushes the branch to ECR from master, or the manually selected branch when invoked manually. We stopped using Docker Hub for storing the image of oplogtoredis, where we used a Webhook to do the same thing.

on:
push:
branches:
- master
# Left this here in-case the workflow needs to be developed further rapidly:
pull_request:
types: [ opened, synchronize, reopened ]
workflow_dispatch:


concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}

permissions:
id-token: write
contents: read

jobs:
build-and-push:
name: Build and Push
runs-on: ubuntu-latest
steps:

- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608

- uses: cachix/install-nix-action@6ed004b9ccb68dbc28e7c85bee15fa93dbd214ac
with:
nix_path: nixpkgs=channel:nixos-unstable

# Examples of the generated tags:
# - v3.0.0 (branch: master)
# - v3.0.0-branch-name (branch: branch-name)
- name: Generate Tag
id: generate-tag
run: |
# Extract the current version from `default.nix`. This must pass.
version=$(nix flake show . --quiet --all-systems --json | jq -r '.defaultPackage."aarch64-darwin".name' | cut -d'-' -f2-)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's fine to hard-code aarch64-darwin ?

Copy link
Contributor Author

@adamtajti adamtajti Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is, cause all the ${system} dicts will have the same .name attribute. Although I agree that it's ugly.


# Adds the branch name if the workflow is manually invoked from a branch.
tag_branch_segment=""

branch_name=${{ github.event.pull_request && github.head_ref || github.ref_name }}
if [ "$branch_name" != "master" ]; then
tag_branch_segment="-${branch_name}" # adds a -<branch-name> as needed
fi

tag="v${version}${tag_branch_segment}"

echo "TAG=${tag}" >> $GITHUB_OUTPUT

- name: Setup ECR/buildx
uses: ./.github/actions/setup-ecr-buildx
id: setup-ecr-buildx
with:
iam-role-to-assume: ${{ secrets.AWS_PUBLIC_ECR_IAM_ROLE_ARN }}
registry-type: public

- name: Build and push image
uses: docker/build-push-action@3b5e8027fcad23fda98b2e3ac259d8d67585f671 # 4.0.0
with:
tags: ${{ steps.setup-ecr-buildx.outputs.ecr_registry }}/tulip/oplogtoredis:${{ steps.generate-tag.outputs.TAG }}
provenance: true
push: true
36 changes: 24 additions & 12 deletions .github/workflows/pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,31 @@ name: Pull Request
on: [ pull_request ]

jobs:
nix_build:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe
with:
go-version: '1.17'
- uses: cachix/install-nix-action@6ed004b9ccb68dbc28e7c85bee15fa93dbd214ac
with:
nix_path: nixpkgs=channel:nixos-unstable
- name: run nix build
run: nix build
lint_and_units:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v3
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe
with:
go-version: '1.17'
- name: setup golangci-lint
run: |
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.42.0
export PATH=$PATH:$(go env GOPATH)/bin
golangci-lint --version
- name: golangci-lint
uses: golangci/golangci-lint-action@3a919529898de77ec3da873e3063ca4b10e7f5cc
with:
version: v1.42
- name: run lint
run: ./scripts/runLint.sh
- name: run unit tests
Expand All @@ -24,34 +36,34 @@ jobs:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- name: run acceptance tests
run: ./scripts/runIntegrationAcceptance.sh
performance:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- name: run performance tests
run: ./scripts/runIntegrationPerformance.sh
fault_injection:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- name: run fault injection tests
run: ./scripts/runIntegrationFaultInjection.sh
meteor:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- name: run meteor tests
run: ./scripts/runIntegrationMeteor.sh
blackbox:
runs-on: ubuntu-20.04
timeout-minutes: 60
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
- name: run blackbox tests
run: ./scripts/runBlackboxTests.sh
1 change: 0 additions & 1 deletion Dockerfile.integration
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ RUN go mod download

WORKDIR /oplogtoredis/integration-tests


FROM integration_base AS acceptance
COPY integration-tests/acceptance/*.go ./acceptance/
WORKDIR ./acceptance
Expand Down
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# oplogtoredis

[![Build Status](https://app.travis-ci.com/tulip/oplogtoredis.svg?branch=master)](https://app.travis-ci.com/tulip/oplogtoredis)
[![Go Report Card](https://goreportcard.com/badge/github.com/tulip/oplogtoredis)](https://goreportcard.com/report/github.com/tulip/oplogtoredis)
[![GoDoc](https://godoc.org/github.com/tulip/oplogtoredis?status.svg)](http://godoc.org/github.com/tulip/oplogtoredis)

Expand All @@ -24,6 +23,20 @@ There are a few things that don't currently work in `redis-oplog` when using the
- Custom namespaces and channels ([`redis-oplog` issue #279](https://github.com/cult-of-coders/redis-oplog/issues/279))
- Synthetic mutations ([`redis-oplog` issue #277](https://github.com/cult-of-coders/redis-oplog/issues/277))

## Nix

We are using Nix at Tulip so we're rolling a `default.nix` and a `flake.nix` wrapper around it for Flake support. The `default.nix` file contains a `vendorHash` attribute, which would be the calculated hash of the downloaded modules to be vendored. To update this value after a `go.sum` file change (which means that different modules were downloaded), refer to the following instructions:

1. Edit `default.nix` and change the `vendorHash` attribute to an empty string then run `nix build`
1. Run `nix build` and observe the output. You are looking for this section:

```text
error: hash mismatch in fixed-output derivation '/nix/store/by8bc99ywfw6j1i6zjxcwdmc5b3mmgzv-oplogtoredis-3.0.0-go-modules.drv':
specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
got: sha256-ceToA2DC1bhmg9WIeNSAfoNoU7sk9PrQqgqt5UbpivQ=
```
1. Use the `sha256-ceToA2DC1bhmg9WIeNSAfoNoU7sk9PrQqgqt5UbpivQ=` for the value of `vendorHash` and verify that `nix build` succeeds.

### MongoDB v5

MongoDB v5 makes a substantial change to the format of the oplog, which makes it much more difficult to assemble the list of changed fields from an oplog entry. Meteor has handled this with a full [oplog v2 to v1 converter](https://github.com/meteor/meteor/blob/devel/packages/mongo/oplog_v2_converter.js), which entails substantial complexity and a high level of dependence on the (undocumented) oplog format, and requires testing against every new version of Mongo. To reduce this maintenance burden, oplogtoredis implements a simplified version of this algorithm that just extracts changed top-level fields.
Expand Down Expand Up @@ -61,9 +74,7 @@ you might use this config:

## Deploying oplogtoredis

You can build oplogtoredis from source with `go build .`, which produces a
statically-linked binary you can run. Alternatively, you can use [the public
docker image](https://hub.docker.com/r/tulip/oplogtoredis/tags/)
You can build oplogtoredis from source with `go build .`, which produces a statically-linked binary you can run. Alternatively, you can use [the public docker image](https://gallery.ecr.aws/tulip/oplogtoredis). Previously we used to host these images in [Docker Hub](https://hub.docker.com/r/tulip/oplogtoredis/tags/). These won't be maintained anymore.

### Environment Variables

Expand Down
27 changes: 19 additions & 8 deletions default.nix
Original file line number Diff line number Diff line change
@@ -1,14 +1,25 @@
{ lib, stdenv, buildGoModule, fetchFromGitHub, installShellFiles }:
buildGoModule {
pname = "oplogtoredis";
version = "2.0.1";

buildGoModule {
pname = "oplogtoredis";
version = "3.0.0";
src = builtins.path { path = ./.; };

vendorSha256 = "sha256-VHiYVJUNtHN2IY4iXZ6kHAa3Avi2VwRH1ySKBrrCDu4=";
postInstall = ''
postInstall = ''
'';
nativeBuildInputs = [installShellFiles];
doCheck = false;
doInstallCheck = false;

# update: set value to an empty string and run `nix build`. This will download Go, fetch the dependencies and calculates their hash.
vendorHash = "sha256-ceToA2DC1bhmg9WIeNSAfoNoU7sk9PrQqgqt5UbpivQ=";

nativeBuildInputs = [ installShellFiles ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a nix linter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I know of.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doCheck = false;
doInstallCheck = false;

meta = with lib; {
description = ''
This program tails the oplog of a Mongo server, and publishes changes to Redis.
It's designed to work with the redis-oplog Meteor package'';
homepage = "https://github.com/tulip/oplogtoredis";
license = licenses.mit;
};
}
6 changes: 3 additions & 3 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ require (
require (
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/go-redis/redis/v8 v8.11.5 // indirect
github.com/golang/protobuf v1.4.3 // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/gomodule/redigo v1.8.5 // indirect
Expand Down
6 changes: 6 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
Expand All @@ -62,6 +64,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/deckarep/golang-set v1.7.1 h1:SCQV0S6gTtp6itiFrTqI+pfmJ4LN85S1YzhDf9rTHJQ=
github.com/deckarep/golang-set v1.7.1/go.mod h1:93vsz/8Wt4joVM7c2AVqh+YRMiUSc14yDtF28KmMOgQ=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
Expand All @@ -78,6 +82,8 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-redis/redis/v7 v7.4.1 h1:PASvf36gyUpr2zdOUS/9Zqc80GbM+9BDyiJSJDDOrTI=
github.com/go-redis/redis/v7 v7.4.1/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
Expand Down
4 changes: 2 additions & 2 deletions integration-tests/acceptance/harness_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"testing"
"time"

"github.com/go-redis/redis/v7"
"github.com/go-redis/redis/v8"
"github.com/kylelemons/godebug/pretty"
"github.com/tulip/oplogtoredis/integration-tests/helpers"
"go.mongodb.org/mongo-driver/mongo"
Expand All @@ -30,7 +30,7 @@ func startHarness() *harness {

// Connect to redis and start listening for the publication we expect
h.redisClient = helpers.RedisClient()
h.subscription = h.redisClient.PSubscribe("*")
h.subscription = h.redisClient.PSubscribe(context.Background(), "*")
h.subscriptionC = h.subscription.Channel()

return &h
Expand Down
4 changes: 4 additions & 0 deletions integration-tests/fault-injection/baseline_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package main
import (
"context"
"testing"
"time"

"github.com/tulip/oplogtoredis/integration-tests/fault-injection/harness"
)
Expand All @@ -16,6 +17,9 @@ func TestBaseline(t *testing.T) {
mongo := harness.StartMongoServer()
defer mongo.Stop()

// Sleeping here for a while as the initial connection seems to be unreliable
time.Sleep(time.Second * 1)

redis := harness.StartRedisServer()
defer redis.Stop()

Expand Down
16 changes: 11 additions & 5 deletions integration-tests/fault-injection/harness/inserter.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,23 @@ func RunInserts(client *mongo.Database, numInserts int, frequency time.Duration)
for i := 0; i < numInserts; i++ {
id := fmt.Sprintf("doc%d", i)

// We set a 50ms timout for the insert: long enough that the insert will
// We set a 100ms timeout for the insert: long enough that the insert will
// succeed if Mongo is working normally, but too short for it to retry during
// a failover
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
// a failover.

// The write may still get through even if the InsertOne call errors out and if the resulting InsertedID is nil.
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := client.Collection("Test").InsertOne(ctx, bson.M{
insertResult, err := client.Collection("Test").InsertOne(ctx, bson.M{
"_id": id,
})

if err != nil {
log.Printf("Warning: mongo insert failed: %s", err)
log.Printf("Warning: mongo insert failed for doc %s: %s", id, err)
if insertResult != nil && insertResult.InsertedID != nil {
log.Printf("Warning: although the previous insert faced this error, the InsertedID wasn't nil, so we'll conclude it was a success (InterruptedDueToReplStateChange). InsertedID: %s", insertResult.InsertedID)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 is there a way to check the type of err (e.g., for InterruptedDueToReplStateChange)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, this worked though, so lets go with this one. There could be many different errors, this was just what I picked up. The result is apparently simply not reliable in case of insert when using replicated mongo.

result = append(result, id)
}
} else {
log.Printf("Inserted doc %s", id)
result = append(result, id)
Expand Down
Loading
Loading