Investigate CI improvements

[castelblanque]

Description:

Current CI solution makes use of CircleCI and GKE, but it is failing at an increasing rate.
Investigate what actions can be taken in order to improve/fix the problems when running CI.
In order to do so, check what other Tanzu projects are doing regarding CI.

Some aspects to take a look at:

• Pipeline tool for running CIs and its potential limitations
• Usage for K8s clusters other than GKE
• Which actions to perform for each push, PR and release
• Possible improvements to the current features of Kubeapps CI
• Independent of external resources such as repos and charts.

Describe the results you expected:

Having a proper CI system that runs seamlessly and allows extension of features.

[castelblanque]

Please find below a list of proposed actions to do for Kubeapps CI. There is no particular order, it is just some initial ideas to discuss and extend/reduce if needed. List is a mix of improvements and fixes.
Goal of this is having a better quality of the product but also making contributors life easier.

1. Run E2E tests for all three browsers available in Playwright when doing a release.
   • Benefit: Covering more browsers to find potential UI issues
   • CI event: On release
2. Add new CI step to check that all files contain the right copyright header (see Cartographer).
   • Benefit: No file in project will be merged in main without Copyright header by mistake
   • CI event: Always
3. Add static code analysis. SonarQube/SonarCloud could be a very good choice.
   • Benefit: Adds continuous inspection, code coverage, code smells detection, code quality gates, etc.
   • CI event: On PRs
4. Clean up of page.waitForTimeout() usage in tests
   • Benefit: Reducing CI tests total time and simplifying tests source code
5. Use pre-commit CLI on dev machines. See example from Pinniped.
   • Benefit: Possibility to add validations even before code is pushed to Git repo (e.g. check Copyright year)
6. (Fix) Install ChartMuseum before Kubeapps in CI, so that initial repos syncing does not fail due to ChartMuseum not being available at the time of Kubeapps installation. Install it in its own namespace, instead of Kubeapps.
   • Benefit: Initial repos syncing is successful and therefore less chance for tests to fail
   • CI event: Always
7. Add additional ChartMuseum instance with public access, so that we can get rid of Gitlab repo and Prometheus repo.
   • Benefit: Not depending on 3rd parties for passing tests
   • CI event: Always
8. Extend tests in GKE: add multicluster, add OIDC
   • Benefit: Multicluster and OIDC scenarios are also tested in GKE
   • CI event: On release
9. (Fix) Get rid of the warn message about difference between Kubectl client and server
   • Benefit: Reduce potential failures due to mismatch in versions
10. Add E2E test cases for Flux and Carvel
    • Benefit: Flux and Carvel scenarios are covered in E2E tests
    • CI event: Always
11. Add VIB as a new step in CI when releasing to test in multiple cloud providers
    • Benefit: Getting Kubeapps teste in multiple K8s flavors
    • CI event: On release
12. Build new versions of integration image from Github actions (already raised in Build and push the integration image during the CI #2792)
    • Benefit: No intervention from developers machine for generating integration images
    • CI event: Manually triggered in CI
13. Encapsulate logic scattered in CircleCI Yaml file into script files (e.g. cluster creation) to ease the transition to Github
    • Benefit: Having all the logic of E2E execution in proper script files instead of the current mix of code embedded in yaml and other code in script files.
14. Generate protoc generated code automatically and remove files from Git repo
    • Benefit: Getting rid of many files to maintain in the repo
    • CI event: Always
15. Unify E2E tests RBAC between OIDC and serviceaccounts. Currently there are 2 users in OIDC or 3 tokens otherwise.
    • Benefit: Same RBAC for all test scenarios (OIDC and serviceaccounts, Kind and GKE)
16. Move CI to Github actions
    • Benefit: Be aligned with most of the Tanzu projects using it
17. Add custom images registry for CI tests with and without authentication
    • Benefit: Tests don't rely on 3rd parties like Docker Hub in order to run

[castelblanque]

Closing as all relevant items have been created as issues.