Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardize e2e testing #1756

Closed
17 tasks done
Tracked by #55
chadwhitacre opened this issue Oct 17, 2022 · 14 comments
Closed
17 tasks done
Tracked by #55

Standardize e2e testing #1756

chadwhitacre opened this issue Oct 17, 2022 · 14 comments
Assignees
@emmatyping

Comments

@chadwhitacre
Copy link
Member

chadwhitacre commented Oct 17, 2022 •
edited by emmatyping
Loading

We have a messy mess of integration testing:

  1. test.sh
  2. integration-test.sh
  3. self-hosted cloudbuild
  4. sentry cloudbuild
  5. snuba cloudbuild
  6. relay cloudbuild

We should standardize this in a way that ensures that e2e is kept in sync across all consumer repos (sentry, snuba, relay) in addition to local environments and self-hosted cloudbuild.

Task list:

  1. Review cloudbuild configs and write new Github Action based on them (needs to handle custom sentry/snuba/relay containers etc)
  2. Add new Github Action to CI pipelines in repos we care about as allow-fail
    1. Try adding end to end tests using new action #1806
    2. Initial self-hosted end-to-end unified CI setup relay#1592
    3. Add self-hosted Sentry end-to-end tests to CI using new Github action snuba#3401
    4. ci: Initial self-hosted end-to-end Github Action sentry#42124
  3. Set up PAT for GCB ⇔ GHA #1867
  4. Add Docker container deployment (set up Dockerhub access token)
    1. relay
    2. snuba
    3. sentry
  5. After two days, make the Action required
  6. Remove old Google Cloud build check
    1. ci: Remove GCB tests and make GHA default for self-hosted e2e tests relay#1721
    2. ci: Remove GCB tests and make GHA default for self-hosted e2e tests snuba#3557
    3. ci: Remove GCB tests and make GHA default for self-hosted e2e tests sentry#42641
    4. Remove GCB and update Github Action SHA #1880
@chadwhitacre chadwhitacre mentioned this issue Oct 17, 2022
Closed
12 tasks
@emmatyping emmatyping mentioned this issue Oct 18, 2022
Closed
5 tasks
@BYK
Copy link
Member

BYK commented Oct 19, 2022 •
edited
Loading

Historical context:

The reason for using Cloudbuild is two fold:

  1. Performance: in the bad old days we did not have GitHub Actions and anything else was too slow or limited in terms of memory (no our build is not demanding :P). Since e2e has been working quite well for some time in self-hosted through GHA, I think that can be revisited. Even more, using hosted-runners for GHA might be a good option too.
  2. The idea was to build the Sentry image and use the same image for single-tenant which lives in GCP. It was a hassle to manage secrets etc, again in the bad old days. Now in the bright future, we have OIDC which should also fix that problem: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-google-cloud-platform

Also, while we are at that, considering switching to GitHub packages instead of Docker Hub may be beneficial both in terms of performance and budget.

Now that I'm thinking, if you switch to using GHA for e2e tests, you can make it a custom action that can be shared across repos where each of them provides the "image override" env variable and you are golden.

@emmatyping
Copy link
Contributor

Thanks for the background @BYK! I agree a custom action seems like the best way to standardize here.

@emmatyping emmatyping mentioned this issue Oct 19, 2022
Closed
@emmatyping
Copy link
Contributor

emmatyping commented Nov 4, 2022 •
edited
Loading

Plan for switching over to the new action: Now in the first comment!

@emmatyping
Copy link
Contributor

Repo is live at https://github.com/getsentry/action-self-hosted-e2e-tests. Planning on adding it to self-hosted today, the rest of the repos next week.

This was referenced Nov 16, 2022
Merged
Merged
@emmatyping emmatyping self-assigned this Nov 18, 2022
emmatyping added a commit to getsentry/relay that referenced this issue Nov 21, 2022
@emmatyping
ci: Initial self-hosted end-to-end unified CI setup (#1592)
6689ca8 
This is work as part of
getsentry/self-hosted#1756

Essentially, we want to move from Google Cloudbuild to a centralized
Github Action that can be used across Sentry/Snuba/Relay/self-hosted.
This adds a job which should use that action.

#skip-changelog
@chadwhitacre chadwhitacre mentioned this issue Nov 22, 2022
Closed
14 tasks
@emmatyping
Copy link
Contributor

All of the actions are now deployed! Need to figure out the docker image deployment situation but once that is done we can start the clock on replacing GCB.

@chadwhitacre
Copy link
Member Author

All of the actions are now deployed! Need to figure out the docker image deployment situation but once that is done we can start the clock on replacing GCB.

FTR we decided to keep Docker build and push in GCB rather than moving to GHA, mostly for security concerns. Now we're trying to kick off the e2e tests in GHA from GCB via a webhook. When we moved GitHub behind Okta I lost access to our bot accounts, so I'm trying to recover that so I can set up a PAT.

@chadwhitacre
Copy link
Member Author

Bot accounts recovered! I created a (new-style) PAT and am looking at setting it up in GCB.

@chadwhitacre
Copy link
Member Author

New-style PATs must expire, going with classic.

@chadwhitacre
Copy link
Member Author

GCB secrets process is complicated ...

@chadwhitacre chadwhitacre mentioned this issue Dec 15, 2022
Closed
6 tasks
@chadwhitacre
Copy link
Member Author

🐰🕳️

@chadwhitacre
Copy link
Member Author

Hit a wall, escalated to Ops.

@chadwhitacre
Copy link
Member Author

chadwhitacre commented Dec 19, 2022 •
edited
Loading

Decided to go with publishing from GHA to Docker after all.

Any concerns about keeping the dockerhub secret secure in Github Actions?

I don't have any concern, no -- I think we should do all the pushing in GHA instead of having to do complicated back and forth between GCB and GHA

I assume only Sentry employees will be able to kick off a release and publish to DockerHub?

The main concern with pushing from Github is people aren't going to check PRs for exposing secrets before hitting the approve button to run CI for PRs from external contributors, so if there is a PR from an outside contributor they might be able to expose the secret

they won't have access to secrets unless they are sentry employees

Even with a PR to getsentry/sentry?

yep. only pull_request_target will have secrets available and we're very particular about what we put into that

I created a token in Docker Hub and set it as a repo secret in sentry, relay, and snuba.

emmatyping added a commit to getsentry/relay that referenced this issue Dec 19, 2022
@emmatyping
Push docker images from self-hosted e2e tests
e808b18 
This enables the e2e test action to push Docker images to DockerHub on successful test runs. This is part of getsentry/self-hosted#1756
This was referenced Dec 19, 2022
Closed
Merged
Merged
emmatyping added a commit to getsentry/sentry that referenced this issue Dec 22, 2022
@emmatyping
Update GHA SHA
9895f09 
This is the final step of getsentry/self-hosted#1756
This was referenced Dec 22, 2022
Merged
Merged
Merged
emmatyping added a commit to getsentry/sentry that referenced this issue Dec 27, 2022
@emmatyping
ci: Remove GCB tests and make GHA default for self-hosted e2e tests (#…
8889fb8 
…42641)

This is the final step of [standardizing self-hosted e2e
tests](getsentry/self-hosted#1756). We remove
tests from Google Cloud Build and replace it with the latest Github Action.
emmatyping added a commit to getsentry/relay that referenced this issue Dec 28, 2022
@emmatyping
ci: Remove GCB tests and make GHA default for self-hosted e2e tests (#…
3b42784 
…1721)

This is the final step of [standardizing self-hosted e2e
tests](getsentry/self-hosted#1756). We remove
tests from Google Cloud Build and replace it with the latest Github
Action.
emmatyping added a commit to getsentry/snuba that referenced this issue Dec 28, 2022
@emmatyping
ci: Remove GCB tests and make GHA default for self-hosted e2e tests (#…
66197e5 
…3557)

This is the final step of [standardizing self-hosted e2e tests](getsentry/self-hosted#1756). We remove tests from Google Cloud Build and replace it with the latest Github Action.
@emmatyping
Copy link
Contributor

This is now done!

@chadwhitacre
Copy link
Member Author

Had to do some cleanup in getsentry/action-self-hosted-e2e-tests#3 ftr.

@github-actions github-actions bot locked and limited conversation to collaborators Jan 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@emmatyping emmatyping

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@BYK @chadwhitacre @emmatyping