feat: Improve Git handler and introduce caching #1689

dominik003 · 2024-08-05T15:37:01Z

Breaking Changes

A new key .Values.valkey.password is required. Fill it with a randomly generated password.

Changes

Persist Git Repository Id: This PR stores the remote repository id (i.e. project id for GitLab and {owner}/{repo_name} for GitHub) along with the git model in the database, so we only need to fetch it once. If the git model path changes, we also need to update the repository id.
Persist Job Identifier for Diagram Cache: This PR allows the job identifier corresponding to the last successful pipeline to be retrieved only once, rather than for each requested diagram. This is achieved by providing the job identifier along with the initial diagram metadata response, so that the frontend can always send the job identifier along with a requested diagram. However, this behavior is optional, so if no job identifier is provided, we will still look up the identifier for each request.
Git Handler Improvements: This PR contains a number of improvements/refactorings to the git handler. For example, the project identifier has been removed from all function parameters and is now set on the object itself, unused functions have been removed, return types have been changed to properly match the actual type, documentation has been added to the abstract git handler, and more.
Git Handler Caching: This PR introduces an initial caching approach for files and artifacts retrieved from git pipelines. This should reduce the number of API calls we make and thus significantly improve image load times. We use redis as a caching solution with the following keys {git path}:{git revision}:f:{file path} for files and {git path}:{git revision}:a:{job id}:{file_path} for artifacts. These keys should uniquely identify a file or artifact, and the hierarchical structure allows us to easily clear the cache for certain git models (i.e., we can remove all keys under {git path}:{git revision}:*).

Resolves #893

TODO:

Add fallback if valkey is not reachable
Mount correct directories to valkey container

codecov · 2024-08-05T16:45:18Z

Codecov Report

Attention: Patch coverage is 70.28302% with 63 lines in your changes missing coverage. Please review.

Project coverage is 84.13%. Comparing base (7e18c2d) to head (9bc25bd).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...jects/toolmodels/modelsources/git/handler/cache.py	35.29%	33 Missing ⚠️
...cts/toolmodels/modelsources/git/handler/handler.py	83.78%	2 Missing and 4 partials ⚠️
...ollab/projects/toolmodels/modelsources/git/crud.py	61.53%	2 Missing and 3 partials ⚠️
...ects/toolmodels/modelsources/git/github/handler.py	79.16%	4 Missing and 1 partial ⚠️
...pellacollab/projects/toolmodels/diagrams/routes.py	70.00%	3 Missing ⚠️
...ects/toolmodels/modelsources/git/gitlab/handler.py	83.33%	3 Missing ⚠️
...cts/toolmodels/modelsources/git/handler/factory.py	88.88%	2 Missing and 1 partial ⚠️
...llacollab/projects/toolmodels/modelbadge/routes.py	50.00%	2 Missing ⚠️
...projects/toolmodels/modelsources/git/validation.py	0.00%	2 Missing ⚠️
...lab/projects/toolmodels/modelsources/git/routes.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1689      +/-   ##
==========================================
- Coverage   84.31%   84.13%   -0.19%     
==========================================
  Files         193      194       +1     
  Lines        6344     6441      +97     
  Branches      693      708      +15     
==========================================
+ Hits         5349     5419      +70     
- Misses        846      867      +21     
- Partials      149      155       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-08-12T13:20:56Z