Code Review Quality Checker.

Motivation. Code review is an important practice for every software team that cares about the quality of its software product. On GitHub, code reviews are usually done within pull requests, where one programmer (reviewer) makes comments asking another programmer (author) to improve the code just submitted in a branch. However, very often, the quality of code review may be rather low: reviewers just say "LGTM" and the pull request gets merged. This GitHub action, with the help of LLMs, analyzes how thorough the code review was and posts a number of suggestions for the reviewer so that they can improve in the future. Besides that, this action suggests "review score," like "excellent review" or "poor review."

Use it like this:

name: code-review
on:
 pull_request_review:
   types: submitted
permissions:
  pull-requests: write
  contents: read
jobs:
  check:
    if: ${{ github.event.review.state == 'approved' }}
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: docker://tracehub/code-review-action:latest
        with:
          openai_token: ${{ secrets.OPENAI_TOKEN }}
          github_token: ${{ secrets.GITHUB_TOKEN }}

In order to skip "too small" pull requests, you can configure min_lines parameter:

name: code-review
on:
 pull_request_review:
   types: submitted
permissions:
  pull-requests: write
  contents: read
jobs:
  check:
    if: ${{ github.event.review.state == 'approved' }}
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: docker://tracehub/code-review-action:latest
        with:
          openai_token: ${{ secrets.OPENAI_TOKEN }}
          github_token: ${{ secrets.GITHUB_TOKEN }}
          min_lines: 15

Code review quality checker can be configured the way you want. These are the parameters you can use/override:

• openai_token: Open AI API key, you can obtain it here.
• github_token: GitHub token in order to post comments in the pull request.
• openai_model: Open AI ChatGPT model, the default one is gpt-4.
• deepinfra_token: Deep Infra API key, you can obtain it here.
• deepinfra_model: Deep Infra API model, check out all available models.
• min_lines: Minimal amount of lines in the pull request to get analyzed by this action, pull requests with fewer lines than provided min_size won't be processed.
• skip_authors: GitHub logins of authors, whose pull requests you want to skip from analyzing. By default, renovatebot and dependabot are ignored.

To analyze code review quality, performed by other programmer, we employ LLM. First we parse GitHub pull request to this format:

[
  {
    "filename": "eo-parser/src/test/resources/org/eolang/parser/packs/add-locators.yaml",
    "additions": 5,
    "deletions": 6,
    "changes": 11,
    "patch": "@@ -12,11 +12,10 @@ tests:\n   - //o[not(@base) and @name='e' and @loc='Φ.org.abc.tt.α2.e']\n   - //o[@base='.hello' and @loc='Φ.org.abc.tt.α2.φ']\n   - //o[@base='e' and @loc='Φ.org.abc.tt.α2.φ.ρ']\n-  - //o[@name='q' and @base='.<' and @loc='Φ.org.abc.q']\n-  - //o[@base='.p' and not(@name) and @loc='Φ.org.abc.q.ρ']\n-  - //o[@base='.^' and not(@name) and @loc='Φ.org.abc.q.ρ.ρ']\n-  - //o[@base='.&' and not(@name) and @loc='Φ.org.abc.q.ρ.ρ.ρ']\n-  - //o[@base='$' and not(@name) and @loc='Φ.org.abc.q.ρ.ρ.ρ.ρ']\n+  - //o[@name='q' and @base='.p' and @loc='Φ.org.abc.q']\n+  - //o[@base='.^' and not(@name) and @loc='Φ.org.abc.q.ρ']\n+  - //o[@base='.&' and not(@name) and @loc='Φ.org.abc.q.ρ.ρ']\n+  - //o[@base='$' and not(@name) and @loc='Φ.org.abc.q.ρ.ρ.ρ']\n eo: |\n   +alias org.abc.foo.b\n   +alias x\n@@ -38,4 +37,4 @@ eo: |\n     [e]\n       e.hello > @\n   \n-  $.&.^.p.< > q\n+  $.&.^.p > q"
  },
  ...
]

Then we parse the all the reviews made by the reviewer in this pull request:

[
  {
    "submitted": "@maxonfjvipon, take a look, please",
    "comments": [
      "h1alexbel: Let's refactor it, since..."
    ]
  },
  ...
]

After all this prepared we instruct LLM to analyze how thorough the code review was. In the end of analysis LLM suggests a review score like "excellent review", "fair review", and "poor review".

The next step is to generate suggestions for the reviewer, on how to improve the code review process in future from his side. To do so, we again ask LLM to conduct in this area.

Fork repository, make changes, send us a pull request. We will review your changes and apply them to the master branch shortly, provided they don't violate our quality standards. To avoid frustration, before sending us your pull request please run full maven build:

mvn clean install -Pjacoco

If you want to run simulation integration tests (annotated with @Tag("simulation")):

mvn clean install -Psimulation \
 -DINPUT_GITHUB_TOKEN=...\
 -DINPUT_DEEPINFRA_TOKEN=...\
 -DINPUT_DEEPINFRA_MODEL=...

For INPUT_GITHUB_TOKEN provide your GitHub token with write permissions to the next repositories:

• tracehubpm/test

For INPUT_DEEPINFRA_TOKEN provide your token from Deep Infra, you can obtain it here. For INPUT_DEEPINFRA_MODEL pick one of the available models.

You will need Maven 3.8+ and Java 17+.