Does Drift help a coding agent fix a real dependency upgrade?

Measure whether Drift lets the same coding agent solve a real dependency migration with less model input and a higher probability of a behaviourally correct fix. Not whether Drift’s prompt is shorter, and not whether its answer reads well.

Suite agent-upgrade-v1 (draft), generated 2026-09-16, Drift 627bfda24d (v0.1.11), agent claude-code 2.1.267 (Claude Code), requested model claude-sonnet-5 at effort high; models the sessions confirmed: claude-sonnet-5 (18 trials). 3 case(s), 3 run(s) per condition. Ecosystems: npm ×3. Case roles: development ×3.

A real repository at a fixed commit, with the dependency already upgraded in its manifest and lockfile and the application code not yet migrated. Each trial materializes a fresh git repository with exactly two commits — before and after the bump — no remote, and no later history, so the maintainer’s eventual fix is not reachable.

The agent receives the task text — the dependency, the two versions, an instruction to fix every incompatibility without reverting, run the appropriate checks, and leave the repository working — with its normal tools: file reads, search, shell, the package manager, the installed package contents. Nothing is withheld and nothing is pre-loaded.

The identical task text, followed by Drift’s own report for that upgrade — the same text drift analyze --markdown --verify prints and the GitHub Action posts: computed breaking changes with evidence, the file and line of every place they reach, the project’s own checks run against the bumped tree. One benchmark-authored sentence introduces it. Same model, effort, tools, permissions, network policy, time limit and budget as the baseline.

Gross agent input tokens: for one agent session, the sum over every model the session used of the provider-reported input_tokens + cache_read_input_tokens + cache_creation_input_tokens, read from the coding agent CLI's own cumulative per-model usage record at the end of the session (Claude Code: the `modelUsage` block of the `result` event in `--output-format stream-json`). Every model turn in the session is included, not only the first prompt. Cache reads are counted at their full token size. "Uncached input tokens" (input_tokens + cache_creation_input_tokens) is reported as a secondary metric.

A trial succeeds only if, after the agent exits: the dependency is still declared at the upgraded version and a fresh install resolves and installs it at that version; every one of the project's own declared checks passes; every hidden regression test — staged into the workspace only after the agent exited — passes; and no case-specific prohibited-workaround rule fires. A session that timed out or errored fails. No partial credit.

Each case carries regression tests that exercise the behaviour the upgrade actually broke. They live outside the workspace and are copied in only after the agent process has exited. A case is admitted only if those tests fail at the start state and pass with a known-correct fix, and the known-correct state is validated twice for determinism.

Real incompatibility; start state reproduces the failure; hidden tests fail before and pass after the reference fix; the dependency remains upgraded in the fixed state; validation is deterministic; the workspace audits clean of hidden material; the task is a legitimate migration. Admitted cases are frozen into a versioned suite by content hash.

Efficiency: per case, the median gross input tokens of valid baseline trials against the median of valid Drift trials; the headline is the median of those case-level reductions, never a ratio of totals. Effectiveness: successful valid trials over valid trials per condition, reported as a percentage-point difference, with the case-level view beside it.

two-level percentile bootstrap: cases resampled with replacement, then each condition's trials within each drawn case resampled with replacement Intervals are reported for the median reduction and the success-rate difference. No trial is rerun because of its result.

A provider outage, an agent that could not launch, or a validation command that could not start excludes the trial with the reason recorded, for either condition. An agent that timed out, errored, or produced a wrong fix is an ordinary failure and is never excluded.

Each trial runs in a fresh temporary git repository built from the case source with exactly two commits (before and after the bump), no remote and no later history, audited to contain no hidden material. Hidden tests, the reference patch and case metadata live outside that directory and are copied in only after the agent process has exited. This is a workspace audit, not an OS sandbox: the private files remain readable elsewhere on the host by a process the agent starts. Package manager caches are shared across trials and conditions.

Counts are `tool_use` blocks in the session event stream, deduplicated by block id. "File reads" are Read calls; "unique files read" are distinct Read paths; "searches" are Grep and Glob calls; "shell commands" are Bash tool invocations, each counted once regardless of how many programs the command line ran; "edits" are Edit, Write, MultiEdit and NotebookEdit calls. Files read through a shell command (cat, sed, head) are not counted as file reads.

Agents are stochastic and results depend on the model, agent version and effort recorded; provider caching shapes the gross token figure; wall clock includes network latency; the suite is dependency migrations only; the share of held-out cases is recorded per result and an early suite has few; passing tests cannot prove the absence of every regression; hidden-test quality varies by case and each case’s tests are in the repository for review.

npm run benchmark:agent:validate, then npm run benchmark:agent -- --suite <suite> --runs 5, then npm run benchmark:agent:aggregate -- --runs <run-id>, npm run benchmark:agent:report and npm run benchmark:agent:verify. Raw per-trial artifacts, including the final diff and every validation output, are under eval/results/agent; the full method is in eval/agent/README.md.