Original reference: Gian Segato, “Quantifying infrastructure noise in agentic coding evals”. This article is an original interpretation, not a translation.

The most misleading thing about Agent Benchmark is not the model score, but the infrastructure noise.

In the past few years, everyone has seen a set of muscle memories when looking at model benchmarks: who is taller, who is ahead by several points, which curve is steeper, and which list has the most credibility.

When this kind of reading habit is brought to agent benchmarks, it is easy to create a dangerous misunderstanding: as long as the scoring mechanism is formal enough, the task set is realistic enough, and the diagrams are clear enough, the difference in scores can naturally represent the difference in system capabilities.

The most important contribution of Anthropic’s “Quantifying infrastructure noise in agentic coding evals” is not to tell you how much a specific benchmark has increased, but to force us to face a fact that many people are unwilling to seriously admit: **agentic coding eval has never been a pure model test. It is essentially a system test in which the model, runtime, resource constraints, tool chain and execution environment jointly participate. **

In other words, when you look at an agent benchmark, what you see is never just “how strong the model is.” You also see: how loose the sandbox is, how generous the resource allocation is, how cruel the kill semantics are, how high the concurrency pressure is, how stable the network is, how healthy the cluster is, and whether there is any extra luck when installing dependencies.

If these conditions are not fully accounted for, then the gap in the rankings will be difficult to reliably explain.

And I think this is exactly where today’s agent benchmark is most misleading.

1. Why does agent eval naturally require more infrastructure conditions than static model eval?

To understand the importance of this article, you must first admit one thing: agent eval and traditional model evaluation are not the same thing.

Most traditional static evaluations are closer to input-output comparison: giving questions, producing answers, and comparing the marked answers or judge results.

In this paradigm, the operating environment is of course also important, but its importance is relatively limited because the environment itself is not deeply involved in the problem-solving process.

But agentic coding eval is different. It requires the agent to actually enter the environment and take action: install dependencies, read and write files, start commands, execute tests, observe errors, make repairs, and verify again.

This means that infrastructure is no longer just a “carrying platform” but part of the mission. Resource headroom, disk speed, network jitter, cache dependencies, process limits, these can all change the direction of a task.

You can even say: In the agent benchmark, the infrastructure is not the background, but half the referee.

2. Why small score differences are particularly alarming: You are probably comparing two environments, not just two agents.

The most stinging part of the article is that it reminds us that some seemingly convincing small differences may not be enough to support the strong narratives commonly seen in the outside world.

For example, everyone’s favorite saying is: A is 2 points better than B, so it is ahead in ability. The new version is 1.8 points higher than the old version, which means that the strategy optimization is effective. A certain model’s stable lead on a certain list means that its agent has the strongest ability.

These statements are not necessarily wrong, but their premise is: **Environment variables are sufficiently controlled. **

In the agent benchmark scenario, this premise often does not hold true at all.

When resources are tight, there will be more infra failures; when resources are wide, the agent will adopt more installation-dependent, more trying, and more “heavy” solutions. So the same model looks like two ability levels in different environments.

The problem behind this is not as simple as “the environment will affect the results”, but:

**Once infrastructure variables are opaque, the spread no longer belongs purely to the model or strategy itself. **

3. The resource headroom changes not only the stability, but also the problem-solving style.

Many people will understand infra noise as “the environment is less likely to cause trouble”. This is certainly true, but it is not complete enough.

I think the real deeper insight in the article is: resource allocation not only affects whether the system will crash, it also affects whether the system dares, is able, and is inclined to choose a certain type of solution.

When resources are tight, the system is forced to be more restrained

When memory, time, and network budgets are tight, it is easier to choose an agent: fewer installation dependencies, fewer large-scale tests, more tendency to directly use standard libraries or lightweight solutions, and fewer heavy intermediate steps.

This may be a good thing for some tasks, but it will significantly reduce the success rate for other tasks.

When resources are wide, the system will be more willing to “do a little more”

Once the environment becomes looser, the agent may start to install more libraries, try more paths, perform higher-cost checks, and adopt more complex combinations.

This sometimes makes the task easier to succeed, but it also means that the benchmark is no longer just testing reasoning, but testing how much operating margin the runtime allows the agent. **

So the resource headroom is not just a stability knob, it is also a strategy space knob. This distinction is critical.

4. What is most easily underestimated is not the number of resources, but how the resources are executed and killed.

One thing I particularly agree with in the article is that it does not stop at the superficial questions of “how much RAM is there” and “how much CPU is there”, but further points out: **Execution semantics are equally important. **

This is particularly like the lesson learned in the world of distributed systems: configuration values ​​themselves do not describe actual behavior, behavior depends on how those values ​​are implemented.

The same nominal 4GB RAM, different platforms may perform completely differently. Because there are many details behind it: whether it is a reservation or a hard cap, whether there is burst space, whether there is a direct kill or buffering when OOM occurs, whether there is contention between concurrent tasks, and whether the sandbox is particularly sensitive to short-term peaks.

From a user perspective, these are like “platform details”; but from a benchmark perspective, they directly determine whether the agent has a chance to complete the task.

This is why I feel more and more that a more serious agent eval report must explain not only “resource quota”, but also “resource semantics”. Otherwise the report is providing configurational illusions, not experimental conditions.

5. How does infrastructure noise pollute the explanatory power of the benchmark?

The worst thing about infrastructure noise is not that it causes fluctuations in results, but that it contaminates the interpretation itself. **

Once noise is not explicitly modeled, the entire benchmark narrative becomes increasingly fragile. You don’t know whether the score change comes from a stronger model, better strategies, looser environment, healthier cluster status, avoiding peak hours, less concurrency, or a more stable network.

Teams, media, product developers, and even researchers often like to extract grand conclusions from these scores.

This creates a very typical industry problem: the data originally only supported “a certain system performs better under certain conditions”, but the narrative finally turned into “a certain model is absolutely leading in the agent scenario”.

This kind of jump already exists in static benchmarks, but it is only more dangerous in agent benchmarks.

6. Why does agent benchmark require disclosure discipline, not just leaderboard aesthetics?

I think the agent benchmark will become more and more complicated in the next year or two, but the real value will not be just a more beautiful leaderboard, but a stricter disclosure discipline.

In other words, not only points are given, but conditions are also given.

A truly quotable, comparable, and reproducible agent benchmark should disclose at least four types of key information.

Resource layer: hardware upper limit, resource enforcement method, concurrent configuration.

Environment layer: sandbox strategy, network conditions, external dependency constraints, task timeout strategy.

Behavior layer: Whether to allow additional installation of dependencies and whether to enable caching.

Statistical layer: Evaluation running time window, number of repeated runs and variance observation.

This information may seem boring, but it determines whether the benchmark is really qualified to be used to make judgments.

Otherwise, many so-called “ranking lists” are essentially just a beautifully visualized experimental snapshot.

7. A mature eval harness must be able to distinguish the source of failure, rather than just counting the total number of failures.

Another implicit methodology in the article that I most agree with is that it pushes us to re-view failure.

In agent eval, failures should never be just a total. What a mature system should care about is: where the failure comes from.

It should be broken down into at least several categories: reasoning failure, planning failure, tool-use failure, environment failure, dependency-install failure, timeout / latency failure, resource-kill failure.

If not broken down, the team will keep optimizing on the wrong layer.

For example: the environment is obviously too tight, but the prompt is being changed crazily; the tool discoverability is obviously poor, but the model is suspected to be not smart enough; the benchmark itself is obviously unstable, but it is explaining agent fluctuations.

The real value of a good eval harness is not just “telling you that you failed”, but “telling you what level failure should be classified into.”

8. Repeated experiments, time windows and confidence awareness will become the basic qualities of agent benchmark.

Traditional static benchmarks often give people the illusion that “running once is enough”, especially in scenarios where the input is fixed and the output judgment is relatively stable, this habit can barely hold.

But agent benchmark is different. It naturally depends on the environment and timing, so I think mature teams in the future must establish a stronger awareness of repeated experiments: repeat the same configuration multiple times, repeat the run in different time windows, observe the variance band instead of just the best run, distinguish extreme values ​​from normal values, and avoid using a single optimal result as a story when reporting externally.

This is actually bringing statistical knowledge back to the engineering site. Sounds basic, but a lot of benchmark narratives these days simply don’t do this.

9. A more complete internal usage process: treat the list as input, not as a conclusion

If I were to give a process suggestion to the team that actually works as an agent, I would write the order of using the public benchmark as follows:

Step 1: Use the external list as input - it helps you narrow down the candidates and decide which models, which runtimes, and which workflows are worthy of the next round of verification.

Step 2: Cut out your real tasks - Do not directly reuse the task structure of the public list, but choose: your most common task types, the failure costs you care about most, your most real resource constraints, and the environmental noise you will actually encounter.

Step 3: Conduct local controlled experiments - Re-verify the candidate solutions under local constraints, paying special attention to: success rate, cost, fluctuation, distribution of failure reasons, and whether it is easier to recover.

Step 4: Do decision mapping - Finally, convert the experimental results into organizational actions: whether to continue the pilot, whether it is worth migrating, whether it is worth expanding its use in a certain workflow, and whether it is only applicable to specific tasks.

This process may seem conservative, but it will help teams avoid a very expensive mistake: treating the list as conclusions rather than clues.

Conclusion: The mature way to use agent benchmark is not to use it to replace judgment, but to use it to force yourself to make better judgments.

If I were to leave this article with one sentence that is most suitable to end with, it would be:

The most valuable thing about agent benchmark is not that it makes a judgment for you, but that it forces you to establish a more mature judgment method than “looking at the list and drawing conclusions”.

I think this is where the infrastructure noise discussion should really be directed. It does not mean that people will never look at the benchmark, but that people finally know: when to look at it, how to look at it, and to what extent they should stop and verify it for themselves.

---

Who should read this

This article is suitable for the following types of readers:

• The engineering/research team that is building or referencing the agent benchmark
• People who want to make agent eval a reproducible system experiment
• Technical leaders who often read the leaderboard and make judgments based on it
• The platform team working on sandbox, runtime, and benchmark infra
• People who have strong doubts about “why the same agent performs so differently in different environments”