Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PR evaluation prompt and link to fine-tuning benchmark documentation #940

Merged
merged 1 commit into from
Jun 3, 2024

Conversation

mrT23
Copy link
Collaborator

@mrT23 mrT23 commented Jun 3, 2024

PR Type

documentation, enhancement


Description

  • Added a link to the evaluation prompt in the fine-tuning benchmark documentation to guide users to the evaluation criteria.
  • Introduced a new evaluation prompt in pr_agent/settings/pr_evaluate_prompt_response.toml for comparing and ranking the quality of two responses to a PR task.
  • Defined guidelines and criteria for evaluating the responses, including how well they follow task instructions, analyze the PR code diff, and prioritize key feedback.
  • Provided an example output format for the evaluation, ensuring clarity and consistency in the evaluation process.

Changes walkthrough 📝

Relevant files
Documentation
index.md
Add link to evaluation prompt in fine-tuning benchmark documentation

docs/docs/finetuning_benchmark/index.md

  • Added a link to the evaluation prompt in the fine-tuning benchmark
    documentation.
  • +1/-0     
    Enhancement
    pr_evaluate_prompt_response.toml
    Add evaluation prompt and guidelines for PR response ranking

    pr_agent/settings/pr_evaluate_prompt_response.toml

  • Introduced a new evaluation prompt for comparing and ranking the
    quality of two responses to a PR task.
  • Defined guidelines and criteria for evaluating the responses.
  • Provided an example output format for the evaluation.
  • +68/-0   

    💡 PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    @codiumai-pr-agent-pro codiumai-pr-agent-pro bot added documentation Improvements or additions to documentation enhancement New feature or request Review effort [1-5]: 2 labels Jun 3, 2024
    Copy link
    Contributor

    PR Review 🔍

    ⏱️ Estimated effort to review [1-5]

    2, because the changes are primarily documentation and configuration updates, which are usually less complex to review than code changes. However, the new TOML configuration is lengthy and detailed, requiring careful attention to ensure all guidelines and criteria are correctly specified.

    🏅 Score

    92

    🧪 Relevant tests

    No

    ⚡ Possible issues

    No

    🔒 Security concerns

    No

    🔀 Multiple PR themes

    No

    @Codium-ai Codium-ai deleted a comment from codiumai-pr-agent-pro bot Jun 3, 2024
    @Codium-ai Codium-ai deleted a comment from codiumai-pr-agent-pro bot Jun 3, 2024
    @mrT23 mrT23 merged commit f8feaa0 into main Jun 3, 2024
    1 check passed
    @mrT23 mrT23 deleted the tr/benchmark branch June 3, 2024 08:39
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    documentation Improvements or additions to documentation enhancement New feature or request Review effort [1-5]: 2
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    2 participants