Panorama of AI programming ability evaluation: from HumanEval to SWE-bench, the evolution and selection of benchmarks

Copyright Statement and Disclaimer This article provides a comprehensive interpretation based on public materials such as HumanEval, SWE-bench, LiveCodeBench, APPS, CodeContests, EvalPlus, and Aider Leaderboards. The copyright of the original text belongs to the respective authors and research institutions.

Original reference

• HumanEval — OpenAI: https://github.com/openai/human-eval
• SWE-bench — Princeton/UCB: https://www.swebench.com/
• LiveCodeBench — UCB/MIT/Cornell: https://livecodebench.github.io/
• APPS — Stanford: https://github.com/hendrycks/apps
• CodeContests — DeepMind: https://github.com/deepmind/code_contests
• Aider Leaderboards — Paul Gauthier: https://aider.chat/docs/leaderboards/

Original Nature This article is not a collection of benchmark-by-benchmark summaries, but rather helps readers build a systematic understanding of benchmarking by juxtaposing multiple sources and refining evaluation dimensions and selection frameworks.

---

Beginning: Benchmarking is not a list, it is a tool to measure boundaries

If you have accepted the judgment in Part 1: Developers must not only be users of AI programming assistants, but also play the role of Coding Mentors, then you will soon encounter the second question: How do you judge the boundaries of AI capabilities?

Just checking whether the output can run once is not enough. Just watching the model manufacturer’s demonstration is not enough. Just looking at community reputation is not enough. AI programming ability is not a single score. It includes at least multiple dimensions such as function generation, algorithm reasoning, code editing, real warehouse repair, test generation, error localization, context utilization, cost and stability.

The value of public benchmarking is here. HumanEval, APPS, CodeContests, SWE-bench, LiveCodeBench, and Aider Leaderboards do not provide readers with a permanent answer to “who is the strongest”, but provide a set of measurement methods: different benchmarks put AI into different task environments to see under what conditions it is reliable and under what conditions it fails.

When reading benchmarks, the most important thing is not to remember a certain list ranking, but to answer three questions:

question	Why is it important
What tasks does this benchmark actually measure?	Also called “code ability”, function generation and real warehouse repair are not the same ability.
What this benchmark doesn’t measure	The uncovered portion cannot be inferred from the score
Can it be transferred to your team scenario?	Public scores can only be used for external reference and cannot replace private eval.

This article will be developed according to this reading. You’ll see how benchmarks evolve from small function questions to real GitHub issues, see which questions different benchmarks are suitable for answering, and see how to transform public benchmarks into the team’s own evaluation set.

Panoramic comparison of six categories of benchmarks

The development path of open benchmarks can be roughly understood as moving from “function-level correctness” to “real engineering tasks.” This is not a simple difficulty upgrade, but a change in the assessment object: from a single function, to algorithm competition questions, to issues in real warehouses, to continuous updates and code editing scenarios.

benchmark	Publisher/maintainer	core mission	Main questions answered	Typical blind area
HumanEval	OpenAI	Complete Python functions based on function signature and description	Whether the model has the ability to generate basic functions	Context, engineering constraints and contamination risks
EvalPlus	community maintenance	Extended HumanEval/MBPP test cases	Are original function questions overestimated by weak tests?	Still mainly focused on function-level tasks
APPS	Stanford	Competition style programming questions	Whether the model has algorithmic reasoning and complex input processing capabilities	Competitive ability does not equal engineering ability
CodeContests	DeepMind	Programming competition questions and multi-language submissions	Can the model handle difficult algorithmic challenges?	Lack of real code base maintenance context
SWE-bench	Princeton/UCB	Fix issues in real open source repositories	Can the model understand the code base and generate verifiable patches?	The environmental cost is high and task distribution still has boundaries
LiveCodeBench	UCB/MIT/Cornell	Continuously collect new questions and evaluate them by time slices	Whether the model is still reliable on new questions	More of a programming question, not equivalent to an enterprise code base
Aider Leaderboards	Paul Gauthier	Code editing in the context of the code base	Is the model suitable for daily editing and multi-language modifications?	Results depend on tool workflow and task settings

This table has an important takeaway: no public benchmark can stand alone as a proxy for “AI programming ability.” Each benchmark measures an aspect. Only by stacking multiple aspects together can you get closer to the boundary of your abilities that you really care about.

HumanEval: The starting point for function-level generation capabilities

The historical significance of HumanEval is that it turns “can the model write code” into a repeatable evaluation question. Each question provides a function signature, documentation string, and a few examples. The model needs to complete the function implementation, and then use hidden tests to determine whether it passes.

It is suitable for answering a very narrow but basic question: whether the model can write an independent function with clear input and output.

design point	Inspiration for readers
The task is small	Basic competency assessment should start with isolable tasks
test automation	Evaluation conclusions must be supported by repeatable testing
Use pass@k	Code generation is random, and a single failure does not mean that it will not happen at all.
Hidden tests	Correctness cannot be judged based solely on public examples

HumanEval also has obvious limitations. The number of questions is limited, the circulation time is long, and the risk of data contamination is high; the tasks are mainly independent functions, which cannot represent the dependencies, status, compatibility, and architectural constraints in the real code base; the language range is also narrow. EvalPlus alleviates the problem of weak original tests by supplementing them with stronger test cases, but does not change the basic nature of function-level evaluation.

For Coding Mentor, the value of HumanEval is not to let you copy 164 questions, but to remind you that the minimum evaluation unit must have a clear contract and automatic acceptance. When you design the eval seed set within the team, you also need to first have a batch of basic tasks with “clear boundaries, low cost, and repeatability” to quickly discover the lower limit of the model.

APPS vs. CodeContests: What algorithm challenges can and cannot test

APPS and CodeContests take assessment to competition-level programming questions. Compared with HumanEval, this type of task usually has more complex questions, a larger input range, stricter time constraints, and relies more on algorithm combination and boundary processing.

They are suitable for answering questions about whether the model is capable of complex reasoning, algorithm selection, and the ability to pass intensive testing.

value	explain
Clearly layered difficulty	Competition questions naturally vary in difficulty such as entry level, competition, and Olympiad.
Test cases are stronger	Many questions have been submitted by a large number of players and polished by boundaries.
Can expose reasoning shortcomings	Complex constraints, extreme value inputs, and combinatorial algorithms can make models more prone to errors.

But engineering readers cannot equate competitive ability with software engineering ability. Competition questions usually seek to provide an AC solution within a limited time. Real projects are more concerned with maintainability, readability, local modification, compatibility, security constraints and long-term evolution. Just because a model is good at competition questions doesn’t mean it can safely change permission logic in your business code; just because a model has an average competition ranking doesn’t mean it can’t help with testing, document changes, or partial refactoring.

Therefore, this type of benchmark is more suitable for “complex reasoning stress testing” rather than as the only basis for enterprise selection.

SWE-bench: From writing code to solving real problems

The key change of SWE-bench is to shift the evaluation from “writing a function” to “fixing an issue in a real warehouse”. After the model gets the issue description and code base context, it needs to locate relevant files, understand the cause of the failure, generate patches, and verify them through testing.

This type of task is closer to real software engineering because it examines multiple capabilities simultaneously:

ability	Reflection in SWE-bench
Requirements understanding	Identify the real issue to be fixed from the issue description
Code navigation	Find the relevant call chain in the multi-file code base
root cause analysis	Distinguish between superficial errors and real failure reasons
minimal fix	Modify necessary locations to avoid large-scale irrelevant refactoring
return to consciousness	Fix the target problem without breaking existing behavior

Its inspiration for Coding Mentor is very direct: if you want to evaluate whether AI can participate in real delivery, you cannot just give it an independent question, you must put it into the code base, issue, test and review constraints.

However, SWE-bench is not a complete image of the production environment. It relies on specific open source projects and reproducible test environments. Task distribution is not equal to your business distribution. The ranking results will also be affected by the agent framework, tool invocation, context strategy and evaluation configuration. When reading SWE-bench, don’t just look at the solution rate, but also look at the task settings, whether it is a Verified subset, whether additional tools are used, whether multiple rounds of repair are allowed, and what types of failed samples are concentrated.

LiveCodeBench: Why time slicing matters

When a public benchmark is repeatedly used for training, parameter tuning, and promotion, it faces data pollution problems. If the model performs well on old questions, it may be because its capabilities have improved, or it may be because it has seen similar questions during training. The important value of LiveCodeBench is to introduce the time dimension into evaluation: it continuously collects newly released questions and observes model performance based on release time slices.

This has two implications for readers.

First, the public score depends on the time. A model that performs well on early questions does not necessarily mean that it is still reliable when facing new questions. For questions after the training deadline, the model has a harder time relying on memory and can only rely more on real reasoning and generation capabilities.

Second, task types should be looked at separately. Code generation, self-healing, code execution prediction, and test output prediction are not the same capabilities. A model may be good at generating solutions, but not good at predicting the results of existing code; it may be good at fixing minor errors, but not good at constructing a complete solution from scratch.

Coding Mentor should also retain time-slicing thinking when doing evaluations within the team. Don’t leave assessment sets unchanged for years, and don’t lump all tasks into one total score. New requirements, recent incidents, newly repaired defects, and recent changes in code specifications should all enter the candidate task pool regularly and become new eval samples after governance.

Aider Leaderboards: Code editing is closer to daily development than building from scratch

A lot of real development work is not to write code from scratch, but to edit existing code: modify an interface, add a test, migrate an API, adjust a configuration, fix a regression. Aider Leaderboards focuses on this editing power.

There are essential differences between code editing tasks and function generation tasks:

Dimensions	Generate from scratch	Code editing
enter	Requirements statement and a little context	Existing files, calling relationships, modification instructions
risk	Generate results may be incomplete	Modifications may break existing behavior
key capabilities	Completion, inference, grammatical correctness	Positioning, local modification, maintaining style, reducing irrelevant diff
Verification method	Unit tests or sample tests	Testing, diff review, regression checking

If your team mainly uses AI for daily code modifications, code editing benchmarks are more valuable than pure generation benchmarks. But it still cannot replace private evaluation, because the editing success rate strongly depends on the tool chain, context injection method, language ecology and task type.

For Coding Mentor, the significance of lists like Aider is to remind the team: Don’t just evaluate “whether you can write”, but also evaluate “whether you can make fewer changes, make changes accurately, and make them verifiable”.

How to choose a benchmark that suits your scenario

When reading benchmarks, it is recommended to start from “what decision do you want to make” rather than “which list is more popular”. Different decisions require different evidence.

your question	Preferred public reference	What else needs to be added internally?
Whether the model has basic code generation capabilities	HumanEval / EvalPlus	Small tasks in the team’s commonly used languages ​​and frameworks
Whether the model can handle complex algorithms and boundaries	APPS / CodeContests	Business-related data structures, performance, and extreme inputs
Whether the model can repair real warehouse problems	SWE-bench / SWE-bench Verified	Team history bugfixes, test escapes and review return samples
Is the model overestimated by old questions?	LiveCodeBench	Recent requirements, recent failures, and recent specification changes
Is the model suitable for daily code editing?	Aider Leaderboards	Local modifications and multilingual tasks in real code bases
Is it possible to purchase or switch tools?	Multiple datum cross-references	Cost, latency, security, permissions, data governance and developer experience

When choosing a baseline, there is a simple rule: a public baseline answers “where does this model fit in the xref”, a private eval answers “can this model make it into our delivery process?” Both issues are important but cannot replace each other.

Indicator selection: Don’t be carried away by a total score

Many lists will give overall scores or pass rates, but what teams really want is an ability profile. The higher the total score, the easier it is to conceal local risks. A model may be strong in function generation but weak in real warehouse positioning; it may also have a high editing success rate, but the cost and delay are not suitable for large-scale use.

index	What to measure	Easily misleading
pass@k	The probability of passing the test under multiple samples	Does not represent code quality, maintainability and security
solve rate	Real task end-to-end resolution rate	Binarization results will conceal the cause of failure
edit success rate	Whether the code editing task was successful	Depends on toolchain and task settings
regression rate	Whether the modification breaks existing behavior	Need a strong enough test set
cost / latency	Single task delivery cost and waiting time	Cannot represent quality alone
human review load	Manual review and rework costs	Requires long-term team records

For the evaluation report for Coding Mentor, it is best not to output just a ranking, but to output a capability profile:

Capability dimension	Evidence to be recorded
functional correctness	Test pass rate, failed use cases, verification results after repair
Contextual exploitation	Whether the correct file is found, whether the call chain and existing constraints are understood
minimal modification	diff range, proportion of irrelevant changes, whether to introduce additional dependencies
quality risk	Concurrency, security, performance, compatibility, and maintainability issues
collaboration cost	Prompt rounds, manual modifications, review return reasons

Such a report has more engineering value than “Model A is 3 points higher than Model B” because it can directly determine which tasks the AI ​​should be deeply involved in, which tasks it should only do drafts on, and which tasks it must quit.

How to convert public benchmarks into team private evals

Public benchmarks don’t directly solve team problems, but they can provide design principles. The team’s internal evaluation set does not need to be large at the beginning. The key is that the tasks come from real work, and each sample can be verified, reviewed, and routed.

An implementable private eval can start from 30 to 60 tasks, covering four types of samples:

Sample type	source	Assessment focus
Small function implementation	Recently completed low-risk requirements	Requirements understanding, implementation integrity, and testing awareness
Bug fixes	Historical bugs, test escapes, and online accident review	Root cause location, minimal repair, regression risk
Test enhancement	Boundary conditions missed in the past	Boundary identification, use case design, assertion quality
code review	review returns, architectural constraint violations, security issues	Risk identification, explanation ability, correction suggestions

Each task must retain at least six categories of information:

Field	effect
Task description	Be clear about what the model solves and what it does not solve
code version	Ensure tasks are reproducible
context boundaries	Describes which files, documents, and tools are allowed access
Acceptance criteria	Define what passes, partial passes, and failures mean
Reference fix or reference feedback	As a basis for manual review and subsequent training of candidates
Sensitive data processing results	Prevent private data, credentials and customer information from entering assessment or training

The goal here is not to copy SWE-bench, but to learn its spirit: put the model into real context and judge the results with verifiable evidence. The eval within the team must be closer to its own language stack, framework, coding standards, release process and risk boundaries.

The correct way to read the list: first look at the task definition, then look at the rankings

As of 2026-03, the model release pace, inference modes, tool calling capabilities, and billing methods are all changing rapidly. No article should take a screenshot of a certain list as a long-term conclusion. A safer approach is to record the method of reading the list.

Assessment entrance	Highlights from the list	Questions suitable to be answered
HumanEval / EvalPlus	Is it close to saturation, is there stronger testing, is there a risk of contamination	Is the model’s basic function generation capability acceptable?
SWE-bench Verified	Issue source, verification method, agent settings, whether additional tools are allowed	Can the model handle real warehouse repairs?
LiveCodeBench	Release time slice, task category, performance after training deadline	Whether the model is still reliable on new questions
Aider Leaderboards	Multilingual editing success rate, cost, contextual strategies	Is the model suitable for daily code editing?
Team private eval	Business language, frameworks, security rules, review bounces and costs	Is the model suitable to enter into your delivery process?

It is recommended that the order of reading the list be fixed at four steps:

step	Judgment problem
Let’s look at the task definition first	What does this list measure and what does it not measure?
Let’s look at the evaluation configuration again	Does the model use tools, agents, additional context, and multiple attempts?
Look at the failure sample again	Does the failure focus on understanding, positioning, implementation, testing or regression?
Finally do a private re-inspection	Can the results be transferred to the team’s real tasks?

If you skip the first three steps and only look at the rankings, it is easy to misread the external list as an internal procurement conclusion.

Capability Boundaries: What Public Benchmarks Collectively Reveal

Looking at these benchmarks together, a more robust judgment can be obtained: AI programming capabilities are sufficient to undertake a large number of local tasks, but it cannot be separated from the task contract, evaluation criteria and acceptance mechanism of human Mentor.

AI is generally better at:

scene	reason
Small functions with clear input and output	Task boundaries are clear and tests are easy to automate
Common algorithms and data structures	Rich training corpus and stable model
Local code editing	The scope of modification is controllable and the feedback cycle is short
Standard error fixes	Error reporting, stack and common repair modes are clearer

AI is often more prone to mistakes:

scene	risk
Fuzzy business semantics	The model is easy to complete assumptions that seem reasonable but do not conform to the business
Cross-module architectural changes	Implicit constraints are easily broken when the context is incomplete
performance sensitive path	Passing correctness does not mean acceptable resource consumption
Security and Permission Boundaries	Small changes may expand access or bypass verification
Low test coverage code base	Without strong validation, errors are harder to detect in a timely manner

This goes right back to the main thread of this series: Benchmarking gives you an external mirror, and Coding Mentor gives you an internal judgment system. Without benchmarks, you can easily choose a model based on your feelings; with only benchmarks, you can easily mistake public scores for production reliability.

Conclusion: The benchmark is the entrance, and private eval is the starting point.

The value of open benchmarks is not to make the final decision for the team, but to help you establish a measurement language. HumanEval tells you how function-level tasks are automatically evaluated, APPS and CodeContests tell you how complex reasoning puts pressure, SWE-bench tells you that real warehouse tasks must enter the context and test closed loop, LiveCodeBench tells you the importance of time slicing and pollution control, and Aider tells you that code editing ability should be measured separately.

But the final work of Coding Mentor is to transform these public experiences into the team’s own evaluation system:

Public benchmarks provided	The team needs to make up
standard task paradigm	Real business tasks and code base context
Automated verification ideas	Team acceptance criteria and review rules
Ability Boundary Reference	Private risk boundaries and usage policies
Model horizontal comparison	Cost, latency, security and governance decisions

So, the conclusion of Part 2 is simple: Don’t be superstitious about lists, and don’t ignore them. The public benchmark is responsible for helping you see the outer boundaries of AI programming capabilities, and the private eval is responsible for answering whether it can enter your project delivery process.

When we next enter Part 3, the problem will be further narrowed: since public benchmarks cannot directly replace team assessment, how should a set of high-quality programming questions and task samples be designed.

---

References and Acknowledgments

• HumanEval — Chen et al., OpenAI: https://github.com/openai/human-eval
• EvalPlus — EvalPlus contributors: https://github.com/evalplus/evalplus
• APPS — Hendrycks et al., Stanford: https://github.com/hendrycks/apps
• CodeContests — Li et al., DeepMind: https://github.com/deepmind/code_contests
• SWE-bench — Jimenez et al., Princeton/UCB: https://www.swebench.com/
• LiveCodeBench — Jain et al., UCB/MIT/Cornell: https://livecodebench.github.io/
• Aider Leaderboards — Paul Gauthier: https://aider.chat/docs/leaderboards/