Set the CI to run and publish benchmark reports once a week

[tatiana]

Dependencies

• Depends on: Change benchmark tool to run in the cloud (GCP) #432
• Depends on: Export the metrics to GCS when running the benchmark on the cloud #434
• Depends on: Change benchmark reporting to expose the information from GCS to Markdown #442
• Depends on one of:
  • Evaluate the performance of loading datasets with Astro Python SDK 0.9.0 into Snowflake #438
  • Evaluate the performance of loading datasets with Astro Python SDK 0.9.0 into Postgres #431
  • Evaluate the performance of loading datasets with Astro Python SDK 0.9.0 into BigQuery #437

Acceptance criteria

• The CI should run the benchmark & analysis scripts on a weekly basis for all scenarios defined
• If any of the benchmark scenarios had a performance degradation of 10% (took 10% of the time more than the previous execution of that same scenario) - we should notify in slack
• The markdown created by the analysis script should be committed to the repo
• Script to compare benchmarking results with previous run

[tatiana]

Just giving additional information, this feature was requested by @vikramkoka during a meeting with him - and the goal is to make sure our benchmark results are always up-to-date with the current performance of the Astro Python SDK.

[pankajastro]

I have started working on this

[pankajkoti]

@pankajastro explored Github Actions and will try out Github action along the lines of this ticket today.

[pankajastro]

There is a scheduled event in GitHub workflow but looks like it runs on the default branch only i.e main our case so we might have to merge PR in order to full end-to-end test. I have tested the scheduling part on my private repo it seems to be working. Currently, trying to test the benchmark make command in separate job once this succeeds then we should be able to combine both together

[pankajastro]

trying to run below job

Run-Benchmark:
    runs-on: ubuntu-18.04
    env:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/google_credentials.json
      GOOGLE_APPLICATION_CREDENTIALS_JSON: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_JSON }}
    steps:
      - uses: actions/checkout@v2
      - run: python -c 'import os; print(os.getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON", "").strip())' > ${{ env.GOOGLE_APPLICATION_CREDENTIALS }}
      - run: cd tests/benchmark && make

Error: Error when reading or editing Project Service --****/container.googleapis.com: Failed to list enabled services for project astronomer-dag-authoring: googleapi: Error 403: Permission denied to list services for consumer container

[pankajastro]

trying to run below job

Run-Benchmark:
    runs-on: ubuntu-18.04
    env:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/google_credentials.json
      GOOGLE_APPLICATION_CREDENTIALS_JSON: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_JSON }}
    steps:
      - uses: actions/checkout@v2
      - run: python -c 'import os; print(os.getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON", "").strip())' > ${{ env.GOOGLE_APPLICATION_CREDENTIALS }}
      - run: cd tests/benchmark && make

Error: Error when reading or editing Project Service --****/container.googleapis.com: Failed to list enabled services for project astronomer-dag-authoring: googleapi: Error 403: Permission denied to list services for consumer container

this permission has been resolved

[pankajastro]

Script failing while pushing image into registry

unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Makefile:37: recipe for target 'container' failed
make: *** [container] Error 1
Error: Process completed with exit code 2.

https://github.com/astronomer/astro-sdk/actions/runs/3067653209/jobs/4954955710

[pankajastro]

Script failing while pushing image into registry

unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Makefile:37: recipe for target 'container' failed
make: *** [container] Error 1
Error: Process completed with exit code 2.

https://github.com/astronomer/astro-sdk/actions/runs/3067653209/jobs/4954955710

This got fixed after adding below step

gcloud auth activate-service-account --key-file $(GOOGLE_APPLICATION_CREDENTIALS)

[pankajastro]

Benchmark script succeeded in CI https://github.com/astronomer/astro-sdk/actions/runs/3092321371/jobs/5003571780
commit: f3d0a7c

[sunank200]

I will help @pankajastro with this task