👨‍💻 Awesome Code LLM

 

🔆 How to Contribute

Contributions are welcome! If you have any resources, tools, papers, or insights related to Code LLMs, feel free to submit a pull request. Let's work together to make this project better!

 

News

• 🔥🔥🔥 [2024-11-12] Qwen2.5-Coder series are released, offering six model sizes (0.5B, 1.5B, 3B, 7B, 14B, 32B), with Qwen2.5-Coder-32B-Instruct now the most powerful open-source code model.
• 🔥🔥 [2024-11-08] OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models is released.

 

🧵 Table of Contents

• 🧵 Table of Contents
• 🚀 Top Code LLMs
• 💡 Evaluation Toolkit
• 🚀 Awesome Code LLMs Leaderboard
• 📚 Awesome Code LLMs Papers
  • 🌊 Awesome Code Pre-Training Papers
  • 🐳 Awesome Code Instruction-Tuning Papers
  • 🐬 Awesome Code Alignment Papers
  • 🐋 Awesome Code Prompting Papers
  • 🐙 Awesome Code Benchmark & Evaluation Papers
• 🙌 Contributors
• Cite as
• Acknowledgement
• Star History

 

🚀 Top Code LLMs

Sort by HumanEval Pass@1

Rank	Model	Params	HumanEval	MBPP	Source
1	o1-mini-2024-09-12	-	97.6	93.9	paper
2	o1-preview-2024-09-12	-	95.1	93.4	paper
3	Qwen2.5-Coder-32B-Instruct	32B	92.7	90.2	github
4	Claude-3.5-Sonnet-20241022	-	92.1	91.0	paper
5	GPT-4o-2024-08-06	-	92.1	86.8	paper
6	Qwen2.5-Coder-14B-Instruct	14B	89.6	86.2	github
7	Claude-3.5-Sonnet-20240620	-	89.0	87.6	paper
8	GPT-4o-mini-2024-07-18	-	87.8	86.0	paper
9	Qwen2.5-Coder-7B-Instruct	7B	88.4	83.5	github
10	DS-Coder-V2-Instruct	21/236B	85.4	89.4	github
11	Qwen2.5-Coder-3B-Instruct	3B	84.1	73.6	github
12	DS-Coder-V2-Lite-Instruct	2.4/16B	81.1	82.8	github
13	CodeQwen1.5-7B-Chat	7B	83.5	70.6	github
14	DeepSeek-Coder-33B-Instruct	33B	79.3	70.0	github
15	DeepSeek-Coder-6.7B-Instruct	6.7B	78.6	65.4	github
16	GPT-3.5-Turbo	-	76.2	70.8	github
17	CodeLlama-70B-Instruct	70B	72.0	77.8	paper
18	Qwen2.5-Coder-1.5B-Instruct	1.5B	70.7	69.2	github
19	StarCoder2-15B-Instruct-v0.1	15B	67.7	78.0	paper
20	Qwen2.5-Coder-0.5B-Instruct	0.5B	61.6	52.4	github
21	Pangu-Coder2	15B	61.6	-	paper
22	WizardCoder-15B	15B	57.3	51.8	paper
23	CodeQwen1.5-7B	7B	51.8	61.8	github
24	CodeLlama-34B-Instruct	34B	48.2	61.1	paper
25	Code-Davinci-002	-	47.0	-	paper

 

💡 Evaluation Toolkit:

• bigcode-evaluation-harness: A framework for the evaluation of autoregressive code generation language models.
• code-eval: A framework for the evaluation of autoregressive code generation language models on HumanEval.

 

🚀 Awesome Code LLMs Leaderboard

Leaderboard	Description
Evalperf Leaderboard	Evaluating LLMs for Efficient Code Generation.
Aider Code Editing Leaderboard	Measuring the LLM’s coding ability, and whether it can write new code that integrates into existing code.
BigCodeBench Leaderboard	BigCodeBench evaluates LLMs with practical and challenging programming tasks.
LiveCodeBench Leaderboard	Holistic and Contamination Free Evaluation of Large Language Models for Code.
Big Code Models Leaderboard	Compare performance of base multilingual code generation models on HumanEval benchmark and MultiPL-E.
BIRD Leaderboard	BIRD contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB. It also covers more than 37 professional domains, such as blockchain, hockey, healthcare and education, etc.
CanAiCode Leaderboard	CanAiCode Leaderboard
Coding LLMs Leaderboard	Coding LLMs Leaderboard
CRUXEval Leaderboard	CRUXEval is a benchmark complementary to HumanEval and MBPP measuring code reasoning, understanding, and execution capabilities!
EvalPlus Leaderboard	EvalPlus evaluates AI Coders with rigorous tests.
InfiBench Leaderboard	InfiBench is a comprehensive benchmark for code large language models evaluating model ability on answering freeform real-world questions in the code domain.
InterCode Leaderboard	InterCode is a benchmark for evaluating language models on the interactive coding task. Given a natural language request, an agent is asked to interact with a software system (e.g., database, terminal) with code to resolve the issue.
Program Synthesis Models Leaderboard	They created this leaderboard to help researchers easily identify the best open-source model with an intuitive leadership quadrant graph. They evaluate the performance of open-source code models to rank them based on their capabilities and market adoption.
Spider Leaderboard	Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

 

📚 Awesome Code LLMs Papers

🌊 Awesome Code Pre-Training Papers

Title	Venue	Date	Code	Resources
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models	Preprint	2024.11	Github	HF
Qwen2.5-Coder Technical Report	Preprint	2024.09	Github	HF
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence	Preprint	2024.06	Github	HF
StarCoder 2 and The Stack v2: The Next Generation	Preprint	2024.02	Github	HF
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence	Preprint	2024.01	Github	HF
Code Llama: Open Foundation Models for Code	Preprint	2023.08	Github	HF
Textbooks Are All You Need	Preprint	2023.06	-	HF
CodeT5+: Open Code Large Language Models for Code Understanding and Generation	Preprint	2023.05	Github	HF
StarCoder: may the source be with you!	Preprint	2023.05	Github	HF
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages	ICLR23	2023.05	Github	HF
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X	Preprint	2023.03	Github	HF
SantaCoder: don't reach for the stars!	Preprint	2023.01	-	HF
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis	ICLR'23	2022.03	Github	HF
Evaluating Large Language Models Trained on Code	Preprint	2021.07	Github	-

 

🐳 Awesome Code Instruction-Tuning Papers

Title	Venue	Date	Code	Resources
Magicoder: Source Code Is All You Need	ICML'24	2023.12	Github	HF
OctoPack: Instruction Tuning Code Large Language Models	ICLR'24	2023.08	Github	HF
WizardCoder: Empowering Code Large Language Models with Evol-Instruct	Preprint	2023.07	Github	HF
Code Alpaca: An Instruction-following LLaMA Model trained on code generation instructions	Preprint	2023.xx	Github	HF

 

🐬 Awesome Code Alignment Papers

Title	Venue	Date	Code	Resources
ProSec: Fortifying Code LLMs with Proactive Security Alignment	Preprint	2024.11	-	-
PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models	Preprint	2024.06	-	-
PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback	Preprint	2023.07	-	-
RLTF: Reinforcement Learning from Unit Test Feedback	Preprint	2023.07	Github	-
Execution-based Code Generation using Deep Reinforcement Learning	TMLR'23	2023.01	Github	-
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning	NeurIPS'22	2022.07	Github	-

🐋 Awesome Code Prompting Papers

Title	Venue	Date	Code	Resources
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging	Preprint	2024.10	Github	-
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step	ACL'24	2024.02	Github	-
SelfEvolve: A Code Evolution Framework via Large Language Models	Preprint	2023.06	-	-
Demystifying GPT Self-Repair for Code Generation	ICLR'24	2023.06	Github	-
Teaching Large Language Models to Self-Debug	ICLR'24	2023.06	-	-
LEVER: Learning to Verify Language-to-Code Generation with Execution	ICML'23	2023.02	Github	-
Coder Reviewer Reranking for Code Generation	ICML'23	2022.11	Github	-
CodeT: Code Generation with Generated Tests	ICLR'23	2022.07	Github	-

 

🐙 Awesome Code Benchmark & Evaluation Papers

Dataset	Title	Venue	Date	Code	Resources
GitChameleon	GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models	Preprint	2024.11	Github	-
Evalperf	Evaluating Language Models for Efficient Code Generation	COLM'24	2024.08	Github	HF
LiveCodeBench	LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code	Preprint	2024.03	Github	HF
DevBench	DevBench: A Comprehensive Benchmark for Software Development	Preprint	2024.03	Github	-
SWE-bench	SWE-bench: Can Language Models Resolve Real-World GitHub Issues?	ICLR'24	2024.03	Github	HF
CrossCodeEval	CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion	NeurIPS'23	2023.11	Github	-
RepoCoder	Repository-Level Code Completion Through Iterative Retrieval and Generation	EMNLP'23	2023.10	Github	-
LongCoder	LongCoder: A Long-Range Pre-trained Language Model for Code Completion	ICML'23	2023.10	Github	-
-	Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation	Preprint	2023.08	-	-
BioCoder	BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models	ISMB'24	2023.08	Github	-
RepoBench	RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems	ICLR'24	2023.06	Github	HF
Evalplus	Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation	NeurIPS'23	2023.05	Github	HF
Coeditor	Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing	ICLR'24	2023.05	Github	-
DS-1000	DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation	ICML'23	2022.11	Github	HF
MultiPL-E	MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation	Preprint	2022.08	Github	HF
MBPP	Program Synthesis with Large Language Models	Preprint	2021.08	Github	HF
APPS	Measuring Coding Challenge Competence With APPS	NeurIPS'21	2021.05	Github	HF

 

🙌 Contributors

This is an active repository and your contributions are always welcome! If you have any question about this opinionated list, do not hesitate to contact me huybery@gmail.com.

 

Cite as

@software{awesome-code-llm,
  author = {Binyuan Hui, Lei Zhang},
  title = {An awesome and curated list of best code-LLM for research},
  howpublished = {\url{https://github.com/huybery/Awesome-Code-LLM}},
  year = 2023,
}

 

Acknowledgement

This project is inspired by Awesome-LLM.

 

Star History

⬆ Back to ToC