Drift

// benchmarks

What Drift scores on other people’s datasets

Five public breaking-change datasets, tested with the same Drift build the CLI ships. The results come straight from the run artifacts.

How to read this page

  • 01The datasets ask two different questions: can Drift identify the upstream change, and can it tell whether a repository is affected?
  • 02Only Roseau accuracy dataset (replication kit) and Towards Better Comprehension of Breaking Changes in the NPM Ecosystem include real negative controls, so only those runs support precision and false-positive rates.
  • 03Each result keeps its own denominator. When the data cannot support a metric, the card says so.

Upstream breaking-change detection

Given what an upstream maintainer published — prose, or two versions of a library — does Drift work out what changed?

Roseau accuracy dataset (replication kit)

mavenSource

On hand-built Java library version pairs with exhaustively enumerated API breaking changes, does Drift produce the same set?

In the dataset
267
Evaluated
267
Scored
255
Excluded
12

ground-truth-contested

157 of the scored cases are negative controls — changes labelled not breaking. They are what make a false positive measurable.

Precision

98/99

99.0%

Recall

98/98

100.0%

F1

0.995

harmonic mean

TP 98 · FP 1 · TN 156 · FN 0

breaking-change detection recall98/98 (100.0%)95% 100–100%

What this does not establish. Nothing about Drift on real-world Java projects: this dataset is constructed to enumerate API change kinds exhaustively, which is what makes precision meaningful and also what makes it unrepresentative of an actual upgrade.

Methodology, provenance and per-slice results

The one corpus here with real negatives. 267 hand-built Java version pairs, each labelled by hand for whether the change is binary-breaking, and roughly half labelled *not* breaking — which is what makes precision, a false-positive rate and F1 defined here and nowhere else in this benchmark.

Drift runs its real Java surface diff: computeSurfaceDiff from the shipped build, routed to the japicmp provider, shelling out to the japicmp binary. The dataset ships source rather than jars, so both trees are compiled with javac and served to production's own fetchArchive as though Maven Central had them. The substitution is transport; every line of analysis between the download and the result is production's.

Scored against the isBinaryBreaking column, because japicmp compares classfiles and binary compatibility is the question it answers. The source-compatibility label is carried in every case artifact for anyone who wants to score it the other way.

The replication kit's own recorded labels for Roseau, Japicmp and Revapi travel in each artifact. They are labels as recorded in the kit, produced on the authors' machines with their tool versions — not re-run here — and any comparison drawn from them should say so.

6 reported change(s) did not belong to any of the 267 labelled case packages; 0 are unexplained. Each remaining unmatched change is classified below rather than silently discarded:

- testing_lib.accessModifierClazzNestedClazzAccessDecreaseProtectedToNon.AccessModifierClazzNestedClazzAccessDecreaseProtectedToNon$Clazz — non-benchmark-source-package: accessModifierClazzNestedClazzAccessDecreaseProtectedToNon is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment. - testing_lib.accessModifierClazzNestedClazzAccessDecreaseProtectedToNon.AccessModifierClazzNestedClazzAccessDecreaseProtectedToNon$Clazz.AccessModifierClazzNestedClazzAccessDecreaseProtectedToNon$Clazz — non-benchmark-source-package: accessModifierClazzNestedClazzAccessDecreaseProtectedToNon is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment. - testing_lib.accessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate.AccessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate$Clazz — non-benchmark-source-package: accessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment. - testing_lib.accessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate.AccessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate$Clazz.AccessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate$Clazz — non-benchmark-source-package: accessModifierClazzNestedClazzAccessDecreaseProtectedToPrivate is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment. - testing_lib.accessModifierClazzNestedIfazeAccessDecreaseProtectedToNon.AccessModifierClazzNestedIfazeAccessDecreaseProtectedToNon$Ifaze — non-benchmark-source-package: accessModifierClazzNestedIfazeAccessDecreaseProtectedToNon is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment. - testing_lib.accessModifierClazzNestedIfazeAccessDecreaseProtectedToPrivate.AccessModifierClazzNestedIfazeAccessDecreaseProtectedToPrivate$Ifaze — non-benchmark-source-package: accessModifierClazzNestedIfazeAccessDecreaseProtectedToPrivate is present in the Roseau source tree but absent from results/bench/jezek_dietrich.json, so it is not one of the 267 labelled benchmark cases and cannot change a per-case TP/FP/TN/FN assignment.

Provenance

Dataset version: 10.5281/zenodo.15536418

Licence: see the replication kit

Citation: Roseau replication kit, Zenodo, DOI 10.5281/zenodo.15536418; tool at https://github.com/alien-tools/roseau

Drift commit: ac9c8b852a

Run date: 2026-09-16

Platform: linux/x64, Node v22.23.2

Run id: roseau-accuracy

Ground truth

Granularity: api-symbol

Exhaustive at that granularity: yes

Metrics it can support: recall, precision, f1, category-accuracy

Constructed library pairs with every API breaking change enumerated at symbol level, which is what supports precision and F1.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • exact: 267

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Slicebreaking-change detection recall
groundTruthSourceBreaking: false20/20 (100.0%)
groundTruthSourceBreaking: true78/78 (100.0%)
label: binary-breaking98/98 (100.0%)

Toolchain the run actually had

  • node: v22.23.2
  • npm: 10.9.8
  • git: git version 2.55.0
  • java: openjdk version "21.0.12.1" 2026-08-18 LTS
  • mvn: Apache Maven 3.9.16 (2bdd9fddda4b155ebf8000e807eb73fd829a51d5)
  • docker: Docker version 28.0.4, build b8034c0
  • python3: Python 3.12.3
  • uv: not installed
  • japicmp: installed (version unknown)

Reproduce

npm run eval:external -- roseau

Artifacts, including every per-case prediction and label, are in eval/results/roseau-accuracy/.

Towards Better Comprehension of Breaking Changes in the NPM Ecosystem

npmSource

Given the real prose an upstream JavaScript maintainer wrote about a commit, does Drift read a breaking change out of it, and does it read the right kind of breaking change?

In the dataset
1511
Evaluated
1511
Scored
1511
Excluded
0

No negative controls. Every case in this corpus is a known breakage, so precision and a false-positive rate are not defined here and are not shown.

breaking-change detection recall1431/1511 (94.7%)95% 94–96%
category classification accuracy

Scored on 134 of 1511 cases — the rest have a label this corpus does not record precisely enough to check (1375 ambiguous, 2 unsupported), not cases that were skipped.

91/134 (67.9%)95% 60–76%

What this does not establish. Nothing about whether Drift finds these changes in a consumer repository, nothing about repair, and nothing about the published-artefact API diff — this corpus supplies prose, so only the prose interpreter is under test.

Methodology, provenance and per-slice results

This is a conditional experiment on one production stage: Drift's prose interpreter (extractBreakingPassages + analyze, imported from the shipped build). The corpus is commit messages — it contains no consumer repositories, no version pairs and no build oracles — so it establishes nothing about detection in a repository, localization, or repair.

Provenance

Dataset version: 10.5281/zenodo.13857646

Licence: CC-BY-4.0

Citation: Dezhen Kong et al., "Towards Better Comprehension of Breaking Changes in the NPM Ecosystem", replication package, Zenodo, DOI 10.5281/zenodo.13857646.

Drift commit: ac9c8b852a

Run date: 2026-09-16

Platform: linux/x64, Node v22.23.2

Run id: kong-rq2-category

Ground truth

Granularity: commit

Exhaustive at that granularity: yes

Metrics it can support: recall, precision, f1, category-accuracy

RQ1's detected_bc_are_documented.csv labels every sampled commit yes/no for whether developers documented a breaking change, so the negatives are real and precision is defined. RQ2's analyzed_breaking_changes.csv annotates a category per breaking change on commits already known to be breaking, which supports category accuracy and nothing else.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • ambiguous: 1375
  • exact: 134
  • unsupported: 2

Deliberately not reported

  • precision — Towards Better Comprehension of Breaking Changes in the NPM Ecosystem contributed no negative/control cases to this run, so there is no population a false positive could be measured against.
  • F1 — F1 is a harmonic mean of precision and recall, and precision is not defined here.
  • false-positive rate — Towards Better Comprehension of Breaking Changes in the NPM Ecosystem contributed no negative/control cases to this run, so there is no population a false positive could be measured against.

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Slicebreaking-change detection recallcategory classification accuracy
label: change_behavior963/1026 (93.9%)—
label: change_signature128/134 (95.5%)91/134 (67.9%)
label: inline2/2 (100.0%)—
label: move class7/7 (100.0%)—
label: move field1/1 (100.0%)—
label: move method12/12 (100.0%)—
label: move module5/5 (100.0%)—
label: move2/2 (100.0%)—
label: remove class26/26 (100.0%)—
label: remove constant10/10 (100.0%)—
label: remove field26/27 (96.3%)—
label: remove interface2/2 (100.0%)—
label: remove method121/126 (96.0%)—
label: remove module16/17 (94.1%)—
label: remove type1/1 (100.0%)—
label: remove2/2 (100.0%)—
label: rename class21/22 (95.5%)—
label: rename constant1/1 (100.0%)—
label: rename field29/29 (100.0%)—
label: rename interface4/4 (100.0%)—
label: rename method48/50 (96.0%)—
label: rename module2/3 (66.7%)—
label: rename package1/1 (100.0%)—
label: rename1/1 (100.0%)—
markerBaselinePredictsBreaking: false0/47 (0.0%)0/5 (0.0%)
markerBaselinePredictsBreaking: true1431/1464 (97.7%)91/129 (70.5%)
messageStatesDetail: false340/391 (87.0%)23/37 (62.2%)
messageStatesDetail: true1091/1120 (97.4%)68/97 (70.1%)

Toolchain the run actually had

  • node: v22.23.2
  • npm: 10.9.8
  • git: git version 2.55.0
  • java: openjdk version "17.0.20.1" 2026-08-18
  • mvn: Apache Maven 3.9.16 (2bdd9fddda4b155ebf8000e807eb73fd829a51d5)
  • docker: Docker version 28.0.4, build b8034c0
  • python3: Python 3.12.3
  • uv: not installed
  • japicmp: not installed

Reproduce

npm run eval:external -- kong

Artifacts, including every per-case prediction and label, are in eval/results/kong-rq2-category/.

Towards Better Comprehension of Breaking Changes in the NPM Ecosystem

npmSource

Given the real prose an upstream JavaScript maintainer wrote about a commit, does Drift read a breaking change out of it, and does it read the right kind of breaking change?

In the dataset
16333
Evaluated
16333
Scored
16333
Excluded
0

16168 of the scored cases are negative controls — changes labelled not breaking. They are what make a false positive measurable.

Precision

161/192

83.9%

Recall

161/165

97.6%

F1

0.902

harmonic mean

TP 161 · FP 31 · TN 16137 · FN 4

breaking-change detection recall161/165 (97.6%)95% 95–99%

Trivial baseline on the same cases

Predicts "breaking" whenever the commit message contains a literal BREAKING CHANGE annotation. Not Drift, and not presented as Drift — it is here so a reader can see how much of this task is reachable without reading the text.

conventional-commits marker — precision 164/193 (85.0%), recall 164/165 (99.4%)

What this does not establish. Nothing about whether Drift finds these changes in a consumer repository, nothing about repair, and nothing about the published-artefact API diff — this corpus supplies prose, so only the prose interpreter is under test.

Methodology, provenance and per-slice results

This is a conditional experiment on one production stage, and its labels are close to trivially recoverable: Kong collected breaking commits by looking for a conventional-commits BREAKING CHANGE annotation, so 164 of the 165 positives contain that literal marker and so do 29 of the negatives. The marker baseline is reported beside Drift's result for exactly that reason. Read the two together; a score a regular expression also achieves is not evidence about a tool.

Provenance

Dataset version: 10.5281/zenodo.13857646

Licence: CC-BY-4.0

Citation: Dezhen Kong et al., "Towards Better Comprehension of Breaking Changes in the NPM Ecosystem", replication package, Zenodo, DOI 10.5281/zenodo.13857646.

Drift commit: ac9c8b852a

Run date: 2026-09-16

Platform: linux/x64, Node v22.23.2

Run id: kong-rq1-documented

Ground truth

Granularity: commit

Exhaustive at that granularity: yes

Metrics it can support: recall, precision, f1, category-accuracy

RQ1's detected_bc_are_documented.csv labels every sampled commit yes/no for whether developers documented a breaking change, so the negatives are real and precision is defined. RQ2's analyzed_breaking_changes.csv annotates a category per breaking change on commits already known to be breaking, which supports category accuracy and nothing else.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • exact: 16333

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Slicebreaking-change detection recall
label: documents-breaking-change161/165 (97.6%)
markerBaselinePredictsBreaking: false0/1 (0.0%)
markerBaselinePredictsBreaking: true161/164 (98.2%)
messageStatesDetail: false47/48 (97.9%)
messageStatesDetail: true114/117 (97.4%)

Toolchain the run actually had

  • node: v22.23.2
  • npm: 10.9.8
  • git: git version 2.55.0
  • java: openjdk version "17.0.20.1" 2026-08-18
  • mvn: Apache Maven 3.9.16 (2bdd9fddda4b155ebf8000e807eb73fd829a51d5)
  • docker: Docker version 28.0.4, build b8034c0
  • python3: Python 3.12.3
  • uv: not installed
  • japicmp: not installed

Reproduce

npm run eval:external -- kong

Artifacts, including every per-case prediction and label, are in eval/results/kong-rq1-documented/.

Consumer impact

Given a real project at a real commit and an upgrade known to break it, does Drift see the update, decide the project is affected, and avoid telling the developer it is safe?

Read these three together, not against each other. All three are run against the same engine build — comparing rates across runs of different vintages is unsound, so they move together whenever the engine does. What they do not support is a ranking: swe-bump-bench and TimeMachine are an order of magnitude smaller than BUMP, so their rates carry intervals wide enough to swallow the differences between them. BUMP is the only one of the three precise enough to argue about, and the Java cards under “Known weaknesses” are where that argument is made.

swe-bump-bench

npmSource

Given a real TypeScript project at a real commit and a dependency upgrade known to break its build, does Drift detect the update, decide the repository is affected, localize the code, and avoid telling the developer it is safe?

In the dataset
63
Evaluated
63
Scored
53
Excluded
10

source-unavailable, reproduction-failed

No negative controls. Every case in this corpus is a known breakage, so precision and a false-positive rate are not defined here and are not shown.

dependency-update detection rate53/53 (100.0%)95% 100–100%
affected-repository identification rate33/53 (62.3%)95% 49–75%
consumer localization rate30/53 (56.6%)95% 43–70%
false-safe verdicts0/53 (0.0%)95% 0–0%

What this does not establish. No precision and no false-positive rate. Every task in this corpus is a known-breaking upgrade, so there is no negative population to compute one over; a tool that answered "affected" unconditionally would score identically on the questions this corpus can answer.

Methodology, provenance and per-slice results

Detection only. Every task here is a known-breaking upgrade — the corpus was collected by bumping packages until tsc failed — so there is no negative population and no precision or false-positive rate can be computed from it. The questions it does answer are whether Drift sees the update, decides the repository is affected, points at consumer code, and above all whether it ever tells a developer with a broken build that they are fine.

Installation and the tsc oracle are not run in this track. These projects pin four different Node majors and three package managers against lockfiles from 2022–2024, and a yarn install that fails in 2026 is a fact about the registry and this machine rather than about Drift. Keeping the oracle separate lets an install failure exclude a case from repair questions without discarding the detection answer.

Provenance

Dataset version: d4504129fd9b536ad5aca2e74f35f5d51e3362e4

Licence: see the repository

Citation: xeol-io, swe-bump-bench, https://github.com/xeol-io/swe-bump-bench

Drift commit: 0de69bc20a (working tree dirty)

Run date: 2026-09-07

Platform: darwin/x64, Node v24.20.0

Run id: swe-bump-full

Ground truth

Granularity: project-build

Exhaustive at that granularity: no

Metrics it can support: recall, repair-success, false-safe-count

Each task is a known-breaking dependency bump whose build oracle fails. Positives only: there are no non-breaking control upgrades, and no symbol-level annotation of what changed.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • exact: 63

Deliberately not reported

  • precision — swe-bump-bench's annotation is not exhaustive at the granularity Drift predicts at (project-build): Each task is a known-breaking dependency bump whose build oracle fails. Positives only: there are no non-breaking control upgrades, and no symbol-level annotation of what changed.
  • F1 — F1 is a harmonic mean of precision and recall, and precision is not defined here.
  • false-positive rate — swe-bump-bench's annotation is not exhaustive at the granularity Drift predicts at (project-build): Each task is a known-breaking dependency bump whose build oracle fails. Positives only: there are no non-breaking control upgrades, and no symbol-level annotation of what changed.

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Sliceaffected-repository identification rateconsumer localization ratedependency-update detection rate
exactVersionAdjudicated: true33/53 (62.3%)30/53 (56.6%)53/53 (100.0%)
label: known-breaking-upgrade33/53 (62.3%)30/53 (56.6%)53/53 (100.0%)
versionToIsRange: false3/3 (100.0%)2/3 (66.7%)3/3 (100.0%)
versionToIsRange: true30/50 (60.0%)28/50 (56.0%)50/50 (100.0%)

Toolchain the run actually had

  • node: v24.20.0
  • npm: 11.12.1
  • git: git version 2.50.1 (Apple Git-155)
  • java: openjdk version "19" 2022-09-20
  • mvn: Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
  • docker: Docker version 29.7.2, build a7dcaa6
  • python3: Python 3.12.14
  • uv: not installed
  • japicmp: installed (version unknown)

Reproduce

npm run eval:external -- swe-bump

Artifacts, including every per-case prediction and label, are in eval/results/swe-bump-full/.

BUMP — reproducible breaking dependency updates in Java

mavenSource

Given a real Java project at the commit where a dependency update broke its Maven build, does Drift detect the update and identify the project as affected?

In the dataset
571
Evaluated
571
Scored
551
Excluded
20

source-unavailable

No negative controls. Every case in this corpus is a known breakage, so precision and a false-positive rate are not defined here and are not shown.

dependency-update detection rate495/551 (89.8%)95% 87–92%
affected-repository identification rate386/551 (70.1%)95% 66–74%
consumer localization rate282/551 (51.2%)95% 47–55%
false-safe verdicts17/551 (3.1%)95% 2–5%

What this does not establish. No precision and no false-positive rate. Every record is a reproduced breaking update; there are no non-breaking control updates in the corpus.

Methodology, provenance and per-slice results

Detection only, and the commit pair is real. BUMP records the commit that changed the version in pom.xml, so its parent is the pre-update state — this run diffs the two revisions a developer actually received rather than a bump the harness applied. Drift reads pom.xml directly, so nothing has to be built for the manifest diff, evidence retrieval and localization to run exactly as they do for a user.

No precision and no false-positive rate: every record is a reproduced *breaking* update, and the corpus contains no non-breaking control updates to measure one against.

Provenance

Dataset version: 324d5513aa5ca40b5cb32de5b816a58fa60bd7bb

Licence: see the repository

Citation: Frank Reyes et al., "BUMP: A Benchmark of Reproducible Breaking Dependency Updates", arXiv:2401.09906; data at https://github.com/chains-project/bump, archive at DOI 10.5281/zenodo.10041883.

Drift commit: 65686d9a77 (working tree dirty)

Run date: 2026-09-07

Metrics recomputed: 2026-09-11 at c9644ed9ba — from the same recorded per-case results; the observations are unchanged

Platform: darwin/x64, Node v24.20.0

Run id: bump-full-571

Ground truth

Granularity: project-build

Exhaustive at that granularity: no

Metrics it can support: recall, repair-success, false-safe-count

Each record is one reproduced breaking dependency update with a recorded failure category. Positives only, and no annotation of which consumer source lines the break lands on.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • compatible: 571

Deliberately not reported

  • precision — BUMP — reproducible breaking dependency updates in Java's annotation is not exhaustive at the granularity Drift predicts at (project-build): Each record is one reproduced breaking dependency update with a recorded failure category. Positives only, and no annotation of which consumer source lines the break lands on.
  • F1 — F1 is a harmonic mean of precision and recall, and precision is not defined here.
  • false-positive rate — BUMP — reproducible breaking dependency updates in Java's annotation is not exhaustive at the granularity Drift predicts at (project-build): Each record is one reproduced breaking dependency update with a recorded failure category. Positives only, and no annotation of which consumer source lines the break lands on.

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Sliceaffected-repository identification rateconsumer localization ratedependency-update detection rate
label: COMPILATION_FAILURE181/220 (82.3%)178/220 (80.9%)214/220 (97.3%)
label: DEPENDENCY_LOCK_FAILURE4/14 (28.6%)0/14 (0.0%)4/14 (28.6%)
label: DEPENDENCY_RESOLUTION_FAILURE0/5 (0.0%)0/5 (0.0%)2/5 (40.0%)
label: ENFORCER_FAILURE67/121 (55.4%)8/121 (6.6%)104/121 (86.0%)
label: TEST_FAILURE133/187 (71.1%)96/187 (51.3%)168/187 (89.8%)
label: WERROR_FAILURE1/4 (25.0%)0/4 (0.0%)3/4 (75.0%)
stratum: no-api-surface-delta71/140 (50.7%)8/140 (5.7%)110/140 (78.6%)
stratum: static-signal-possible315/411 (76.6%)274/411 (66.7%)385/411 (93.7%)

Toolchain the run actually had

  • node: v24.20.0
  • npm: 11.12.1
  • git: git version 2.50.1 (Apple Git-155)
  • java: openjdk version "19" 2022-09-20
  • mvn: Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
  • docker: Docker version 29.7.2, build a7dcaa6
  • python3: Python 3.12.14
  • uv: not installed
  • japicmp: installed (version unknown)

Reproduce

npm run eval:external -- bump

Artifacts, including every per-case prediction and label, are in eval/results/bump-full-571/.

TimeMachine-bench (human-verified subset)

pypiSource

Given a real Python repository whose historical dependency state no longer resolves or runs, does Drift detect the dependency change and identify the repository as affected?

In the dataset
100
Evaluated
100
Scored
68
Excluded
32

no-dependency-update, reproduction-failed

No negative controls. Every case in this corpus is a known breakage, so precision and a false-positive rate are not defined here and are not shown.

dependency-update detection rate66/68 (97.1%)95% 93–100%
affected-repository identification rate39/68 (57.4%)95% 46–69%
consumer localization rate33/68 (48.5%)95% 37–60%
false-safe verdicts0/68 (0.0%)95% 0–0%

What this does not establish. No precision and no false-positive rate, for the same reason as swe-bump-bench: the corpus is migration failures, so every case is a positive.

Methodology, provenance and per-slice results

Detection only, on a manifest pair this harness assembles — and that construction is the weakest in this benchmark, so it is stated on every case rather than left to be inferred. The repository is fetched at its exact historical commit; the after-state repins the requirements the project itself declares to the versions the dataset records as resolving at the migration date. It is *not* built from patch or gold_patch, which are the developer's migration and are ground truth — reading the upgrade out of the fix would hand Drift the answer and call the result detection. Only declared requirements are repinned; the rest of the resolved set is the transitive closure, and writing it into a manifest would invent declarations the project never made.

No precision and no false-positive rate: every task is a migration that fails, and the corpus contains no migrations that succeed.

Not installed here: `uv`. TimeMachine's own reproduction is a container per task built against a date-filtered PyPI index, which is what preserves the historical resolution each task depends on. Without it the oracle was not run and no repair question was asked — those outcomes are absent from every case record rather than recorded as failures, because Drift was never asked and did not fail.

Provenance

Dataset version: 9928dbf1af1405433d2c2e40227f39fd831d3863

Licence: see the repository

Citation: Tohoku NLP, TimeMachine-bench, https://github.com/tohoku-nlp/timemachine-bench

Drift commit: 0de69bc20a (working tree dirty)

Run date: 2026-09-07

Metrics recomputed: 2026-09-11 at 8aa0690ba8 — from the same recorded per-case results; the observations are unchanged

Platform: darwin/x64, Node v24.20.0

Run id: timemachine-full

Ground truth

Granularity: project-build

Exhaustive at that granularity: no

Metrics it can support: recall, repair-success, false-safe-count

Human-verified migration failures. Positives only; no non-failing control migrations.

Label mapping coverage

This corpus’s vocabulary is not Drift’s. Only exact and compatible mappings are scored for category correctness; ambiguous and unsupported ones are counted here rather than forced into the nearest Drift kind.

  • compatible: 100

Deliberately not reported

  • precision — TimeMachine-bench (human-verified subset)'s annotation is not exhaustive at the granularity Drift predicts at (project-build): Human-verified migration failures. Positives only; no non-failing control migrations.
  • F1 — F1 is a harmonic mean of precision and recall, and precision is not defined here.
  • false-positive rate — TimeMachine-bench (human-verified subset)'s annotation is not exhaustive at the granularity Drift predicts at (project-build): Human-verified migration failures. Positives only; no non-failing control migrations.

Breakdown

Every rate again, split by the dataset’s own labels. A pooled figure hides both directions of the interesting result, so it is never the only number here.

Sliceaffected-repository identification rateconsumer localization ratedependency-update detection rate
label: migration-failure-easy20/43 (46.5%)17/43 (39.5%)41/43 (95.3%)
label: migration-failure-hard2/2 (100.0%)2/2 (100.0%)2/2 (100.0%)
label: migration-failure-medium17/23 (73.9%)14/23 (60.9%)23/23 (100.0%)

Toolchain the run actually had

  • node: v24.20.0
  • npm: 11.12.1
  • git: git version 2.50.1 (Apple Git-155)
  • java: openjdk version "19" 2022-09-20
  • mvn: Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
  • docker: Docker version 29.7.2, build a7dcaa6
  • python3: Python 3.12.14
  • uv: not installed
  • japicmp: installed (version unknown)

Reproduce

npm run eval:external -- timemachine

Artifacts, including every per-case prediction and label, are in eval/results/timemachine-full/.

What these runs did not measure

Missing data and a zero score are different things. These questions were outside the runs above.

Repair, on every external corpus

These runs test detection, not repair. Repair needs each dataset’s own test environment, including BUMP’s containers and swe-bump-bench’s installed tsc run. Those environments were unavailable, so repair results are absent rather than counted as failures.

Anything involving a coding agent

No agent provider was configured, so the agent tier never ran. The harness records that as “not attempted,” which keeps it out of the success rate.

Any general claim about an ecosystem

Each result is over the cases named on its card, from one corpus, evaluated on one machine on one date. “Drift is X% accurate on Java” does not follow from any of them, and is not claimed.

Known weaknesses these runs found

The misses are as useful as the scores. Here is where Drift struggled, and what caused it.

Java localization trails TypeScript and Python

The three consumer-impact datasets ask the same question: did Drift find that the project was affected?

TypeScript
33/53 affected
Python
39/68 affected
Java
386/551 affected

Update detection is strong in all three. The gap appears later, when Drift maps a Java API change back to consumer code.

Part of that gap is not a localization miss at all. BUMP records why each build broke, and its build-policy and dependency-resolution failures change no API, so there is nothing for any static analysis to find. On the failure classes that do leave a signal — compilation, test and -Werror failures — Drift identified the affected project in 315/411 cases; on the rest, in 71/140. The Java figure above pools both, because that is the like-for-like comparison with the TypeScript and Python corpora.

On Java, Drift still calls some broken upgrades safe — and Java is the only ecosystem measured well enough to say so

Over BUMP’s Java consumer breakages in full — all 571 records, of which 551 scored — Drift detected the update in 495/551 cases and still returned a false-safe verdict for 17/551 of them (3.1%, 95% CI 1.9–4.9%).

It is also the only rate here precise enough to argue about. The npm and Python consumer corpora are an order of magnitude smaller, and their intervals are correspondingly wide — reaching 6.8% and 5.3% respectively, both far above Java’s. Ranking the ecosystems against each other from these three numbers would be reading noise: what separates them is how much evidence each has, not how well Drift does.

Two of the three causes behind it are fixed. A raised minimum JDK was invisible: japicmp reports it as CLASS FILE FORMAT VERSION on every recompiled class, which Drift’s line grammar did not match, so a jOOQ upgrade requiring Java 17 read as having no incompatible change at all. Signatures were also cut at the first parenthesis, discarding the parameter types that carry a javax to jakarta migration — a break that lands on the consumer’s own import and names nothing the changed library owns.

What remains is mostly the class no static analysis reaches: a behavioural change with no signature change and no changelog sentence describing it. Drift finds those in prose where upstream wrote them down, flags the call sites it can see, and refuses to claim a search that found nothing proves anything. And drift upgrade installs nothing on this verdict alone — the unattended batch requires the project’s own checks to have run against the upgrade and passed.

Drift's prose rules generalise beyond changelog phrasing, with real limits

On 1,511 human-annotated real npm breaking changes, Drift reads a breaking change out of the maintainer’s own commit message in 94.7% of cases — 97.4% where the message says more than the bare marker, 87.0% where it doesn’t. Naming the right kind of change is the harder question: 91/134 of the scoreable cases (67.9%).

The ceiling there is structural, not a rule the corpus is waiting for: about a fifth of the scoreable cases carry no description at all beyond the bare marker, and prose alone cannot name a kind for a sentence that says nothing. Closing the rest needs the published-artefact API diff, a different Drift capability than the prose interpreter this corpus tests.

Whether japicmp is installed changes the Java result completely

Without it, Drift cannot compute a Java API surface, correctly declines to conclude anything, and returns insufficient-evidence. With it, the same cases produce hundreds of API changes. That is honest behaviour rather than a bug — but it means a Java user without japicmp gets far less than these numbers suggest, and every run records which situation it was in.

How to check any of this

Each card’s detail block carries the exact command that produced it. The artifacts are in the repository under eval/results/: a manifest naming the Drift commit and dataset version, the deterministic selection, the probed environment, one JSON line per case with its provenance and Drift’s prediction, the metrics, and every exclusion with its reason.

The datasets are not vendored — they belong to their authors and are cited on each card. benchmarks/README.md has the fetch commands and checksums.

What the harness measures, and the longer argument for what it refuses to measure, is in eval/README.md.

Newest result on this page: 2026-09-16. Every figure is read from the run artifacts in the repository, not written into this page.