Benchmark run retry-20260925-agy-prescriptive-task11-large-discover-baseline-quota
This page contains exactly the publishable benchmark observations recorded in run retry-20260925-agy-prescriptive-task11-large-discover-baseline-quota; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: task-11-mixed-sink-api-migration
Target: agy/gemini-3.8-flash/high
prefer_discover_semedit prompt · prescriptive MCP instructions · unspecified semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | agy/gemini-3.8-flash/high |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | unspecified |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit