can-ai-code: Self-evaluating interview for AI coders

[irthomasthomas]

• the-crypt-keeper/can-ai-code: Self-evaluating interview for AI coders

Title: the-crypt-keeper/can-ai-code: Self-evaluating interview for AI coders

A self-evaluating interview for AI coding models, written by humans and taken by AI.

Key Ideas

• Interview questions written by humans, test taken by AI
• Inference scripts for all common API providers and CUDA-enabled quantization runtimes
• Sandbox enviroment (Docker-based) for untrusted Python and NodeJS code validation
• Evaluate effects of prompting techniques and sampling parameters on LLM coding performance
• Evaluate LLM coding performance degradation due to quantization

News

• 2023-01-23: Evaluate mlabonne/Beyonder-4x7B-v2 (AWQ only, FP16 was mega slow).
• 2

Suggested labels

{ "label-name": "interview-evaluation", "description": "Self-evaluating interview for AI coding models", "repo": "the-crypt-keeper/can-ai-code", "confidence": 96.49 }