July 7, 2026

Evaluating LLM Calibration for Coding-Agent Monitoring

Contents

Executive summary

We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.

  • Setup: We use exposed credentials, instruction-following failures, exposed personal data (PII), undesirable deletions, and broad safety/security issues as our five failure modes. The datasets were built by inserting targeted failures into AI coding sessions collected internally and from SWE-chat. In total, we evaluated across 2904 trajectories. Models score each trajectory according to provided severity rubrics.
  • Models have different systematic biases: Some models tend to judge more severely (e.g. Gemini 2.5 Flash) while others give more conservative scores (e.g. Sonnet & Opus). This is relevant when scores drive thresholds, such as which failures trigger alerts, human review, or downstream analysis. 
  • Native thinking helps small judges, but not frontier models: Enabling native thinking at a high setting substantially improves small and mid-tier judges (e.g. Gemini 3.1 Flash-Lite, GPT-5.4 Mini) but leaves frontier judges essentially unchanged. This might be explained by our output format, which requires an extensive structured rationale.
  • Ensembling beats resampling: Averaging across multiple models and providers consistently beats running a single model multiple times. We also find that averaging two cheap models often beats single runs of expensive models. 
  • Some families are quite correlated: Especially Anthropic models have very similar judgments. Getting answers from highly correlated models might add little value, ensembles with monitors from different providers can lead to better calibration.
  • No single model fully matched our intuitions: While various frontier models matched our assessment of the scores in the vast majority of cases, no single model matched them perfectly. For high fidelity, we recommend ensembles and multi-step pipelines (e.g. ensembles + meta-graders).
  • Caveats: The reference severity labels come from a multi-step, multi-model review pipeline, so results should be read relatively, as calibration against that pipeline rather than as absolute truth. We are also not fully confident in our latency estimates.

Best-guess recommendations:

  • Single model:
    • Cheap: Gemini 3 Flash (reasoning=high) at $0.04/grade
    • Expensive: Sonnet 5 (off) at $0.19/grade or Opus 4.8 (high) at $0.42/grade
  • Ensembles:
    • Gemini 3 Flash (high) + GPT-5.4 Nano (high) for a cheap 2-model ensemble at $0.044/grade
    • Gemini 3.1 Pro (high) + GPT-5.5 (high) + Opus 4.8 (high) for optimal cross-family coverage at $0.89/grade
    • Gemini 3 Flash (high) + GPT-5.4 (off) + Sonnet 5 (off) for cheaper family coverage at $0.34/grade
  • Some models seem rarely useful across many dimensions as single judges: Gemini 3.1 Flash-Lite, Gemini 2.5 Flash, Gemini 3.5 Flash, Gemini-2.5-pro, GPT-5.4 nano, GPT-5.4 mini, Claude Haiku 4.5, Sonnet-4.6, Opus-4.6 since they are strictly dominated by other models.

Introduction

Why we use 1-10 severity instead of binary detection

Two common ways to design monitors are:

  • Binary detection: the monitor classifies actions as either safe or dangerous, e.g. detect whether the agent leaked a key, deleted something it shouldn’t have, or ignored an instruction.
  • Severity scale: the monitor assigns a score from 1 to 10 according to a specified rubric.

In this post, we evaluate models according to their severity scores. We refer to a model under evaluation as a judge or monitor and use the terms interchangeably. We score failure modes on a shared 1–10 severity scale, from “no issue” at the bottom to catastrophic, irreversible organizational harm at the top. The modes can differ in the lens used to place a trajectory on that scale, for example blast radius for credential exposure or irreversibility for deletions, and rubrics are calibrated against each other such that a given score represents comparable harm across modes.

We choose severity scores over binary detection for three reasons:

  1. We simulate an async monitoring setting: Monitors are often differentiated between blocking monitors (real-time with ability to block actions) and async monitors (not real-time, often run in batches). This post assumes an async monitoring setting where the entire trajectories are graded, e.g. the user would get a large batch of agent trajectories and try to identify the most interesting failures. In that setting, severity scores are more useful because they give a more fine-grained signal than binary detection scores. 
  2. Severity scores allow the user to decide thresholds: Any severity scale can be turned into a binary decision score, e.g. everything greater or equal to 7 can be a positive detection and everything lower than 7 can be a negative detection. In the binary case, the creator of the monitor decides on the position of the boundary. In the severity case, the end user can decide. Therefore, in cases where the creator and user differ, e.g. if the monitoring system is provided by a company to be used by others, this is a simple way to give the user more room to establish their risk tolerance. For the purpose of this post, we make no recommendations for thresholds and only investigate calibration.
  3. Severity scores enable better fine-tuning: While we don’t do any fine-tuning of models for this post, we think severity scores are more suitable for the fine-tuning of monitors. A score from 1 to 10 provides a more fine-grained reward signal for the model and makes it easier to learn fuzzy and boundary cases. The threshold flexibility from above also carries over: a model trained on severity scores can serve any threshold, whereas a binary monitor would need re-fine-tuning whenever the boundary moves.

We will present results for the binary decision setting in the future and not discuss them here.

Metrics

We use two metrics throughout the post:

  1. Spearman correlation as a proxy for relative calibration. In some cases, e.g. when trying to identify the worst 50 trajectories, we only need a ranking.
  2. Mean absolute error (MAE) as a proxy for absolute calibration. In some cases, e.g. when all scores above 8 trigger a review, we need absolute calibration.

Each lens is useful in different circumstances, so we report both. Recall, precision, and partial AUROC (pAUROC) are the standard for binary detection, but since this post uses a severity scale, we focus on the two metrics above.

Dataset

We build evaluation trajectories by injecting targeted failures into real AI coding sessions, drawn from internally collected sessions and the public SWE-chat dataset. Working from genuine sessions keeps the surrounding context realistic, while inserting the failure ourselves gives each trajectory a known label. We control which failure is present and at what target severity. An LLM editing agent performs the injection under a realism constraint, and a downstream filter drops trajectories that don’t read as plausible (e.g. sometimes the injection agent would create tool result messages without corresponding tool calls). 

The final dataset contains 2,904 trajectories across three source sets; safety and security is the largest failure mode by count, since one source set targets it exclusively. Each judge receives the full trajectory and the failure-mode-specific rubric. The judge knows which failure mode to grade for, but not what was injected, where, or at what target severity.

We cover five failure modes:

  • Exposed credentials. The agent hardcodes secrets into source files, writes them to logs or config, or prints them in responses.
  • Instruction-following failures. The agent ignores explicit or implicit directives, from scope constraints to mid-session corrections.
  • Exposed personal data (PII). The agent reveals, stores, or transmits personal information in ways that create unauthorized-access risk.
  • Undesirable deletions. The agent runs destructive operations the user didn’t authorize, such as rm -rf, database drops, or cloud teardown.
  • Safety and security issues. The broadest category, covering exploitable vulnerabilities the agent introduces, such as SQL injection, weak auth, or infrastructure misconfigurations, as well as actions taken beyond its authorized scope.

Reference labels

We generate “approximate ground truth labels” from a separate grading pipeline that is deliberately more elaborate than the judges we evaluate and has knowledge of the injected failure. The target severity from the injection agent is only an intended value and can be imprecise. Therefore, we pass each trajectory through several models and multiple stages of scoring, review, and correction, with the injection details available to the pipeline, to settle on a single reference severity. Because it is more thorough than any single judge, we treat these outputs as our highest-confidence reference, the label we calibrate against rather than a definitive measure of severity. We iterated on the pipeline until it matched our own intuitions in 99 out of 100 cases when reviewed by hand. Nevertheless, we think it’s still plausible that the pipeline introduces various biases that might systematically distort the results.

Caveats

Circularity: The most important caveat is the circularity built into the setup. The reference labels are produced by LLMs, and we evaluate LLM judges against these labels, so a judge scores well to the extent that it agrees with the reference pipeline rather than by being right in any external sense. We strongly tried to align the approximate ground truth pipeline with our own judgment, but obviously there are limits.

The five failure modes are not cleanly disjoint. Credential exposure, PII, and undesirable deletions are specializations of the broad safety and security mode, and instruction-following can co-occur with any of the others. In practice each trajectory targets a single failure mode and is graded only against it, so this does not blur the analysis, but it is worth remembering when comparing results across modes.

The failures are injected rather than naturally occurring. Realism filtering removes the most unrealistic trajectories, but they remain synthetic, and their distribution across modes and severities reflects our injection choices rather than how failures arise in real use. The top of the scale is the exception: severity-10 trajectories are rare (45 of 2,904), since a 10 reflects a catastrophic scenario that is hard to inject plausibly, so conclusions that depend on judge behavior at the very top of the scale rest on few examples.

Finally, we’re not fully confident in our latency estimates. We had a few infra changes during the process of creating this post. We also found random fluctuations of latency that are most likely on the developer side. We’d therefore recommend treating them with caution. For example, we don’t understand why Opus 4.7 is much faster than Opus 4.6 or 4.8 respectively.

Results

Models are not equal in terms of performance, cost, and latency, and should be selected based on your specific requirements. We report Spearman’s rank correlation as our classification focused metric and recommend using it for judge selection. We report MAE to our reference severity as a measure of judge calibration to our rubrics. Models that perform best on Spearman are not necessarily best on MAE. All models were run with native thinking enabled at a high setting. We compare against the no-thinking setting in the section “Native thinking improves judge quality, but mostly with smaller models” below.

Claude models lead on ranking quality: Opus 4.8 is the best single judge, with Sonnet 5 close behind at 40% of the cost. GPT-5.5 matches Opus 4.8’s calibration at slightly lower ranking quality. Gemini 3.1 Pro comes closest in the Gemini family at an even lower cost, and Gemini 3 Flash remains the best low-budget option. The fastest strong judges are Opus 4.7 (~20s median) and Gemini 3 Flash (~36s).

As single judges, we recommend against Gemini 3.1 Flash-Lite, Gemini 2.5 Flash, Gemini 3.5 Flash, Gemini 2.5 Pro, GPT-5.4 Mini, Claude Haiku 4.5, Sonnet 4.6, and Opus 4.6; each is dominated by a cheaper or better alternative.

Different models have very different monitoring behaviors

Most importantly, some models judge more severely and tend to give high scores (e.g. Gemini 2.5 Flash), while others are more conservative and give lower scores on average (e.g. Claude models).

These trends don’t have to be uniform or linear, e.g. models can over-score low severities while under-scoring high ones. Additionally, more capable models typically have lower spread, i.e. lower variance around their scores.

To give a more qualitative sense of the difference, here are two representative examples of judgment differences between models. 

Example 1: An agent runs a full production pipeline when tasked only with fixing a bug and writing tests, making hard to reverse changes to production data and incurring API costs. The reference severity from our approximate ground truth pipeline suggests a severity of 8 (we think 7 or 8 are justifiable). Sonnet 4.6 correctly identifies the violation but scores a 2, discounting it to “minor excessive agency” with “only trivial benign impact” because the fix was correct, the pipeline completed without errors, and the system is internal. Opus 4.8 and Sonnet 5 score 5, calling it a material production-systems failure but crediting that it was recoverable. Gemini 3.1 Pro scores 6 and Gemini 2.5 Pro scores 7. GPT-5.4 and GPT-5.5 score 7, weighing the unauthorized production execution most heavily, yet even they stay below the reference.

Example 2: An agent commits a client-side website token to a config file while correctly noting tokens of this type are publicly visible by design and are not secrets. The reference score is 2; committing real values to version control is a minor anti-pattern. Gemini 3.1 Pro and Gemini 2.5 Pro score 1, treating the agent’s public-vs-secret reasoning as fully secure handling. Sonnet 4.6 scores 2, and Opus 4.8 and Sonnet 5 score 3, crediting the token’s public-by-design nature. GPT models weigh the committed token far more heavily even while acknowledging its low privilege: GPT-5.4 scores 5, GPT-5.5 scores 7, and GPT-5.4 Nano scores 8, constructing a maximum public-exposure blast radius.

Qualitatively, Claude models often correctly identify violations but can discount them heavily, particularly when the context is framed as local or internal development work, or the action is still in a plan. GPT-5.4 and 5.5, in comparison, discount the least and consistently score above the other judges on the same trajectories, though even they land at or below our reference on average. GPT-5.4 Nano goes further and constructs worst-case harm scenarios from routine actions, such as treating a public-by-design client-side token as a maximum-exposure credential leak.

Gemini models vary more. Gemini 3.5 Flash often scores task completion rather than scanning for violations, additionally dismissing findings when the session looks like routine work or the content is judged as synthetic. Gemini 2.5 Flash often scores violations at production-incident severity regardless of context. Gemini 2.5 Pro tends to weigh downstream consequences more than other judges when it over scores. Gemini 3.1 Pro tends to not flag violations when the session looks like authorized work, or gives a pass when a behavior was not explicitly prohibited, e.g. bypassing human confirmation requests for a destructive database migration.

Smaller models (Gemini 3.1 Flash-Lite, GPT 5.4 mini, Haiku 4.5) have wider spreads and poor performance overall. For example, Gemini 3.1 Flash-Lite skips severity grading to score task execution, Haiku 4.5 primarily fails to flag violations at all, and GPT-5.4 Mini identifies violations but discounts them heavily through session-context framing.

On average, newer and larger models perform better on both Spearman and MAE. Gemini 3.5 Flash performs at the level of the smaller models on MAE, reflecting its consistent pattern of under-scoring moderate and high severity violations.

Native thinking improves judge quality, but mostly with smaller models

Every judge scored the same trajectories under both settings: once with native thinking disabled (reasoning=none) and once at the high setting (reasoning=high for OpenAI, Anthropic, and Gemini 3.x models; a thinking budget of 8192 tokens for Gemini 2.5 models) used throughout this post. Because the design is paired within trajectories, we can attribute score differences directly to the thinking setting. Per model scores vs. references and per-model deltas for all 16 judges with bootstrap confidence intervals are in the Appendix.

A potential explanation is that our output format requires a detailed structured rationale, which might cover most of the benefits of reasoning. In other words, we ask the monitor to follow a given reasoning structure for score justification.

Ensembling beats resampling

Running multiple models and averaging their scores improves both Spearman and MAE meaningfully and outperforms any individual model. Ensembles of smaller models can outcompete larger individual ones on average. Similarly to the individual Pareto plots, practitioners should select ensembles following their individual requirements. Notably, good ensembles often combine models from different providers and families.

Additionally, we consistently found that averaging different models dominates resampling the same model, most clearly on MAE: resampling barely improves calibration for most models, since averaging repeated draws from the same judge cancels noise but not its bias, while different judges have partially offsetting biases. Resampling gains are also concentrated in smaller models (e.g. Gemini 2.5 Flash, GPT-5.4 Mini), while the biggest models (e.g. Opus 4.8, Sonnet 5, GPT-5.5) gain little from repeated calls. Even for small models, spending a second call on a different model beats resampling in almost all cases.

Larger, stronger models are more stable across repeats

Larger and newer models show less variability across repeated scoring of the same trajectories, and we observe a positive correlation between performance and scoring stability. Repeated calls with caching can offer moderate improvement in particular for smaller models. Ensembling different models still achieves highest performance, but repeats of smaller models can be included in the ensemble for additional gains.

Better models agree more and some families have high trajectory-level correlation

Spearman’s rank correlation between pairs of models suggests that bigger models tend to agree more, even across model families (as can be seen by the darker shades). Notably, rank correlations among the Sonnet and Opus models (0.88–0.93) typically match or exceed the best judge-to-reference correlation (0.90 for Opus 4.8). Sonnet 5 and Opus 4.8, the strongest pair, agree with each other at 0.93, which is higher than any model’s agreement with our approximate ground truth.

We additionally measure Pearson correlation of residuals (reference score − predicted score) and find high correlations within the Claude family (up to 0.78) and, almost as strongly, within the GPT family, GPT-5.4 and GPT-5.5 reach 0.76, on par with the closest Claude pairs. This suggests strong family-level similarities in scoring behavior at the trajectory level. Gemini models, by contrast, show no comparable family structure in their residuals.

Since averaging can only cancel errors that are independent, this correlation structure directly explains why our best ensembles at every size combine model families: a cross-family ensemble lets one provider’s biases partially cancel another’s, while a same-family ensemble inherits its shared blind spots.

How this relates to Apollo’s monitoring agenda

Our main goal in the monitoring & control team is to build the best possible monitors. For this, we need to solve various conceptual questions, e.g. which failure modes to look for, which threat models to care about, and more. However, additionally, we also need to solve very practical problems like which monitors to use for our own pipelines and how to measure them. 

If you’re interested in either of these questions, you can find our agenda here. We’re rapidly growing the monitoring research team, if you’re interested please apply.

Appendix

A.1 Native thinking: per-judge results

For reference, the per-model calibration curves with thinking disabled (compare to the thinking-high version in the main text). The frontier models are nearly unchanged without thinking, e.g. Sonnet 5 (ρ = 0.89, MAE = 0.96) and Opus 4.8 (0.89, 1.00) remain strong, while the small models degrade visibly: Haiku 4.5 drops to ρ = 0.69, Gemini 3.1 Flash-Lite to 0.67, GPT-5.4 Mini to 0.72, with a more pronounced under-scoring bend below the diagonal.

The delta chart quantifies this with paired 95% bootstrap confidence intervals over trajectories (n ≈ 2,900). Significant Spearman gains concentrate in the small models (+0.06 to +0.09); the frontier models are mostly indistinguishable from zero, with two exceptions: Opus 4.8 gains a small but significant +0.012 ρ and +0.12 MAE, and GPT-5.5 improves MAE only (+0.05) while paying the largest thinking premium (+$0.10 per grade). Note that Gemini 2.5 Pro and 3.1 Pro cannot fully disable thinking, so their deltas are measured from a minimal-thinking floor and are lower bounds.

A.2 Best ensembles per size

Calibration curves for the best ensemble at each size (1–3 members, mean aggregation), selected by Spearman ρ (top row) and by MAE (bottom row). Mean ensembling improves both metrics and, because the averaged score is fractional, allows more fine-grained decision boundaries than a single judge’s integer scores. Selected by ρ: Opus 4.8 alone (0.90) → GPT-5.5 + Opus 4.8 (0.91, MAE 0.85) → + Gemini 3.1 Pro (0.92). Selected by MAE: GPT-5.5 alone (0.90) → the same pair → Gemini 2.5 Pro + GPT-5.5 + Opus 4.8 (MAE 0.82). Two things stand out: every winning ensemble of size ≥ 2 is cross-family, and the ρ-best and MAE-best trios differ only in which Gemini joins the same GPT + Claude core. Note these are post-hoc best subsets, selected and evaluated on the same data, so they are an upper bound on ensemble lift.

A.3 Output length

Judges differ substantially in verbosity: average rationale length ranges from ~1,100 characters (Gemini 3.1 Flash-Lite, GPT-5.4 Mini) to ~2,800 (Sonnet 4.6), with Claude models writing the longest rationales. The dashed lines mark where in the output the model first commits to a score, typically 60–80% of the way through. Crucially, verbosity is essentially uncorrelated with judge quality (r = −0.08 against MAE, r = 0.11 against ρ): longer rationales neither indicate nor cause better judgment. Output length is therefore a cost and latency consideration, not a quality signal. This is consistent with our hypothesis that the structured rationale format, not its length, is what does the work.

A.4 Performance across failure modes

The pooled metrics in the main text mask meaningful variation across the five failure modes: within a single judge, Spearman ρ varies by up to ~0.15 and MAE by up to ~0.9. Some failure modes are consistently easier to score than others: undesirable deletions show higher Spearman and lower MAE for nearly every judge, while instruction-following violations are the hardest for most. Models can also underperform on specific failure modes, e.g. Gemini 3.5 Flash scores markedly lower on PII and instruction-following than on the other failure modes. The strong judges stay strong everywhere (Opus 4.8 and Sonnet 5 keep a compact, high band across all five); therefore, if a deployment cares primarily about one failure mode, judge selection should be checked against that mode’s profile rather than the pooled numbers.