Benchmark run gemini37-quota-probe-20261006
This page contains exactly the publishable benchmark observations recorded in run gemini37-quota-probe-20261006; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: task-00-hi-overhead
Target: agy/gemini-3.7-flash-medium
explicit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-00-hi-overhead |
| Target | agy/gemini-3.7-flash-medium |
| Prompt variant | explicit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_00_hi_overhead.txtar |
Vanilla LLM Prompt:
Do not modify any workspace files.
Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not modify any workspace files.
Reply hi. Do not inspect or modify the workspace; no semantic tools are needed. Parallelize independent read-only lookups. Perform mutations serially in the stated order or inside one ordered semantic_batch; never run overlapping mutations concurrently. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
| Metric | Vanilla (Small) | ⚠ Semedit Explicit (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 4.99s | 4.55s | N/A | — | — | — |
| Process Start → First Event | — | — | — | — | — | — |
| First Event → First Tool Call | — | — | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | N/A | — | — | — |
| Internal Tool Cycles | 1 | 1 | N/A | — | — | — |
| Initial Load / Discovery Turns | 0 | 0 | — | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 0 | 0 | — | — | — | — |
| Output Tokens | 165 | 227 | N/A | — | — | — |
| Reasoning / Thinking Tokens | 162 | 224 | N/A | — | — | — |
| Total Input Tokens | 16397 | 16448 | N/A | — | — | — |
| Cached Input Tokens | 0 | 0 | — | — | — | — |
| Uncached Input Tokens | 16397 | 16448 | N/A | — | — | — |
| Cost | — | — | — | — | — | — |
| Cached vs Uncached Token Ratio | 0.00:1 | 0.00:1 | — | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | FAIL | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | none recorded | none recorded |
Semedit integration feedback (agent opinion, excluded from measurements):
Because this particular run only required a simple conversational reply without touching any code, I did not actually invoke or evaluate `semedit`'s tools, so I cannot honestly tell my owner whether they are worth adopting yet.
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
implicit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-00-hi-overhead |
| Target | agy/gemini-3.7-flash-medium |
| Prompt variant | implicit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_00_hi_overhead.txtar |
Vanilla LLM Prompt:
Do not modify any workspace files.
Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not modify any workspace files.
Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
| Metric | Vanilla (Small) | ⚠ MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 4.99s | 7.27s | N/A | — | — | — |
| Process Start → First Event | — | — | — | — | — | — |
| First Event → First Tool Call | — | — | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | N/A | — | — | — |
| Internal Tool Cycles | 1 | 1 | N/A | — | — | — |
| Initial Load / Discovery Turns | 0 | 0 | — | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 0 | 0 | — | — | — | — |
| Output Tokens | 165 | 115 | N/A | — | — | — |
| Reasoning / Thinking Tokens | 162 | 112 | N/A | — | — | — |
| Total Input Tokens | 16397 | 16403 | N/A | — | — | — |
| Cached Input Tokens | 0 | 0 | — | — | — | — |
| Uncached Input Tokens | 16397 | 16403 | N/A | — | — | — |
| Cost | — | — | — | — | — | — |
| Cached vs Uncached Token Ratio | 0.00:1 | 0.00:1 | — | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | FAIL | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | none recorded | none recorded |
Semedit integration feedback (agent opinion, excluded from measurements):
Because this specific task strictly prohibited modifying or inspecting the workspace, I did not have the opportunity to exercise `semedit`'s tools firsthand, so I cannot honestly assess their real-world impact or justify asking my owner to add them based on this run alone.
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |