Your Evals Are Green. That Is Not the Same as Working.

Likelier than most teams assume, and the mechanism is duller than people expect. A 2026 audit of code reinforcement-learning environments sampled 49 SWE-bench Verified tasks and found 28.5% of them had test suites weak enough that a Docker-verified incorrect patch passed. Not a model rewriting its own tests, simply tests that fail to reject a wrong answer. The same study measured the consequence across 134 frontier models: scores on hackable tasks ran 14.14 percentage points above robust tasks at matched difficulty, which is capability you did not gain. If a sample of a task set that many people have read has that much slack in it, an internal suite written under deadline is not in better shape.

Often yes, and the failure mode depends on what the judge is doing, which is a distinction worth being strict about. Where a judge acts as a reward signal inside a training loop, you get optimization against it: judge scores climb while real capability plateaus, and the winning strategies are frequently cosmetic, such as padding output with brackets and markup. Where a judge is a static evaluator, the problem is bias rather than optimization: position effects, a preference for longer answers, and a preference for output resembling the judge model itself. Both are real, they are not the same failure, and the fixes differ. What makes either worse is a single scalar score, because it conflates several quality dimensions into one number that moves with whichever is easiest. Rubrics that score dimensions separately are more work and considerably harder to game.

The honest answer splits by who is asking. For a product team shipping agents, those tools and the ecosystem around them, including OpenAI Evals and the UK AI Safety Institute's Inspect, are mature enough that writing your own runner is rarely the best use of a quarter. For a lab running rollouts at scale, custom runners are normal and usually necessary, because distributed execution and hard isolation of untrusted agent output are not what a general-purpose framework optimizes for. What no framework in either category does is design your tasks, decide what a correct answer is, calibrate difficulty, or tell you whether a grader can be shortcut. That is where suites are won and lost, and it is not a tooling problem.

A task set is only informative in a band. If the model under test solves everything, the set cannot rank anything and it will saturate the moment a better model arrives. If it solves nothing, you learn only that the tasks are hard, not which capability is missing. Calibration means running rollouts, measuring where pass rates land, and adjusting task difficulty until the set discriminates. It gets skipped because it is slow, it happens before any headline result exists, and it is the least visible work in the pipeline. It is also the difference between an eval that informs a decision and one that produces a number.

No, and if volume annotation is what you need, the large data vendors are a better fit and we will point you at them. We work on the engineering above that layer: environment design, grader construction, adversarial review, calibration and the readiness standard that decides what ships. That is a small-team job that benefits from one senior person who has built it before, rather than from a bench.

Not independently, and we would not claim otherwise. The same separation that applies in model risk applies here: whoever builds the grader should not be the only party certifying that the grader is sound. Where independence matters to your governance or to a customer, we build and harden, and a second party assesses. Where it does not, we will still run the adversarial pass against our own work and show you what it found, because the exercise is worth doing regardless of who signs it.

An eval audit on an existing suite is 2-3 weeks. Building a calibrated task set with hardened graders in a new domain takes longer and depends almost entirely on how contested the definition of a correct answer is. Domains where correctness is checkable by execution move fast. Domains where it is a judgment call move at the speed of the rubric, and the rubric is the project.