Benchmark run task11-gpt-6-luna-medium-mcp-matrix-repeats10-20260927
This page contains exactly the publishable benchmark observations recorded in run task11-gpt-6-luna-medium-mcp-matrix-repeats10-20260927; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: task-11-mixed-sink-api-migration
Target: codex/gpt-6-luna/medium
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 41.69s | 90.98s | +118.2% | 45.52s | 122.03s | +168.1% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.14s | — |
| First Event → First Tool Call | 7.30s | 6.03s | — | 6.28s | 10.81s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 14 | +180.0% | 6 | 23 | +283.3% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 14 | +180.0% | 6 | 23 | +283.3% |
| Output Tokens | 1533 | 1581 | +3.1% | 1256 | 2514 | +100.2% |
| Reasoning / Thinking Tokens | 298 | 611 | +105.0% | 387 | 883 | +128.2% |
| Total Input Tokens | 120725 | 277682 | +130.0% | 163035 | 639278 | +292.1% |
| Cached Input Tokens | 107008 | 245760 | +129.7% | 129280 | 594688 | +360.0% |
| Uncached Input Tokens | 13717 | 31922 | +132.7% | 33755 | 44590 | +32.1% |
| Cost | 0.0839 | 0.1686 | +100.9% | 0.1372 | 0.3026 | +120.5% |
| Cached vs Uncached Token Ratio | 7.80:1 | 7.70:1 | -1.3% | 3.83:1 | 13.34:1 | +248.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Go verification on path '.' performs formatting that may write across the entire workspace, including forbidden test files; use a narrower non-mutating check instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): read Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518256069894000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518256069894000/audit: is a directory |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518256069894000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned by the delivery b |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_organize_imports (✓ ✓) |
| 17 | — | ? ✗ |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 38.74s | 62.55s | +61.5% | 36.99s | 80.72s | +118.2% |
| Process Start → First Event | 0.13s | 0.24s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 5.47s | 5.16s | — | 5.31s | 8.17s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 10 | +25.0% | 6 | 16 | +166.7% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 10 | +25.0% | 6 | 16 | +166.7% |
| Output Tokens | 1071 | 1187 | +10.8% | 1066 | 1251 | +17.4% |
| Reasoning / Thinking Tokens | 290 | 384 | +32.4% | 97 | 284 | +192.8% |
| Total Input Tokens | 141245 | 273821 | +93.9% | 140329 | 425856 | +203.5% |
| Cached Input Tokens | 128256 | 252160 | +96.6% | 126208 | 391936 | +210.5% |
| Uncached Input Tokens | 12989 | 21661 | +66.8% | 14121 | 33920 | +140.2% |
| Cost | 0.0815 | 0.1368 | +67.8% | 0.0814 | 0.2020 | +148.1% |
| Cached vs Uncached Token Ratio | 9.87:1 | 11.64:1 | +17.9% | 8.94:1 | 11.55:1 | +29.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_organize_imports (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | ? ✗ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 25.06s | 49.82s | +98.8% | 50.71s | 71.70s | +41.4% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.14s | 0.12s | — |
| First Event → First Tool Call | 4.44s | 6.53s | — | 6.05s | 6.30s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 12 | +140.0% | 7 | 12 | +71.4% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 12 | +140.0% | 7 | 12 | +71.4% |
| Output Tokens | 623 | 745 | +19.6% | 1311 | 1410 | +7.6% |
| Reasoning / Thinking Tokens | 0 | 0 | — | 470 | 450 | -4.3% |
| Total Input Tokens | 137259 | 287755 | +109.6% | 183989 | 192719 | +4.7% |
| Cached Input Tokens | 125184 | 252928 | +102.0% | 168704 | 169216 | +0.3% |
| Uncached Input Tokens | 12075 | 34827 | +188.4% | 15285 | 23503 | +53.8% |
| Cost | 0.0693 | 0.1596 | +130.4% | 0.1027 | 0.1243 | +21.1% |
| Cached vs Uncached Token Ratio | 10.37:1 | 7.26:1 | -29.9% | 11.04:1 | 7.20:1 | -34.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | ? ✗ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Go verification over path "." may write formatting changes across the workspace, including forbidden test files; a narrower source-only verification is needed. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocke |
| 12 | — | ? ✗ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 36.35s | 64.52s | +77.5% | 46.85s | 82.96s | +77.1% |
| Process Start → First Event | 0.11s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 4.80s | 6.75s | — | 5.78s | 6.81s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 15 | +200.0% | 6 | 14 | +133.3% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 15 | +200.0% | 6 | 14 | +133.3% |
| Output Tokens | 1104 | 1461 | +32.3% | 1219 | 1672 | +37.2% |
| Reasoning / Thinking Tokens | 313 | 307 | -1.9% | 434 | 686 | +58.1% |
| Total Input Tokens | 139589 | 278277 | +99.4% | 163145 | 397620 | +143.7% |
| Cached Input Tokens | 126208 | 250624 | +98.6% | 148480 | 358656 | +141.6% |
| Uncached Input Tokens | 13381 | 27653 | +106.7% | 14665 | 38964 | +165.7% |
| Cost | 0.0827 | 0.1539 | +86.0% | 0.0944 | 0.2165 | +129.3% |
| Cached vs Uncached Token Ratio | 9.43:1 | 9.06:1 | -3.9% | 10.12:1 | 9.20:1 | -9.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ? ✗ |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 100.91s | 68.15s | -32.5% | 42.87s | 84.03s | +96.0% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 17.25s | 4.72s | — | 4.49s | 6.33s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 11 | +120.0% | 8 | 15 | +87.5% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 11 | +120.0% | 8 | 15 | +87.5% |
| Output Tokens | 996 | 1494 | +50.0% | 1349 | 1329 | -1.5% |
| Reasoning / Thinking Tokens | 329 | 335 | +1.8% | 462 | 381 | -17.5% |
| Total Input Tokens | 139918 | 356392 | +154.7% | 205027 | 245064 | +19.5% |
| Cached Input Tokens | 127232 | 332288 | +161.2% | 189952 | 204032 | +7.4% |
| Uncached Input Tokens | 12686 | 24104 | +90.0% | 15075 | 41032 | +172.2% |
| Cost | 0.0801 | 0.1662 | +107.5% | 0.1078 | 0.1750 | +62.3% |
| Cached vs Uncached Token Ratio | 10.03:1 | 13.79:1 | +37.5% | 12.60:1 | 4.97:1 | -60.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_insert_function (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515922651390000/service/dispatch.go: symbol not found |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ? ✗ | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 49.38s | 53.54s | +8.4% | 44.58s | 124.23s | +178.7% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.47s | 6.73s | — | 4.78s | 6.64s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 11 | +37.5% | 7 | 19 | +171.4% |
| Initial Load / Discovery Turns | 2 | 3 | +50.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 11 | +37.5% | 7 | 19 | +171.4% |
| Output Tokens | 1724 | 1105 | -35.9% | 1439 | 2686 | +86.7% |
| Reasoning / Thinking Tokens | 377 | 145 | -61.5% | 318 | 1211 | +280.8% |
| Total Input Tokens | 183506 | 347366 | +89.3% | 163924 | 389090 | +137.4% |
| Cached Input Tokens | 167680 | 322304 | +92.2% | 148480 | 357888 | +141.0% |
| Uncached Input Tokens | 15826 | 25062 | +58.4% | 15444 | 31202 | +102.0% |
| Cost | 0.1077 | 0.1589 | +47.4% | 0.0977 | 0.2162 | +121.3% |
| Cached vs Uncached Token Ratio | 10.60:1 | 12.86:1 | +21.4% | 9.61:1 | 11.47:1 | +19.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | — | ✓ ✓ |
| 10 | — | ? ✗ |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ? ✗ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518577767135000/service/dispatch.go: symbol not found |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 39.31s | 63.00s | +60.3% | 47.49s | 84.92s | +78.8% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.14s | 0.50s | — |
| First Event → First Tool Call | 6.03s | 4.54s | — | 5.54s | 7.12s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 11 | +83.3% | 8 | 12 | +50.0% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 11 | +83.3% | 8 | 12 | +50.0% |
| Output Tokens | 1127 | 1197 | +6.2% | 1505 | 1285 | -14.6% |
| Reasoning / Thinking Tokens | 0 | 317 | — | 347 | 382 | +10.1% |
| Total Input Tokens | 161840 | 270720 | +67.3% | 184967 | 276292 | +49.4% |
| Cached Input Tokens | 147456 | 247040 | +67.5% | 161536 | 244736 | +51.5% |
| Uncached Input Tokens | 14384 | 23680 | +64.6% | 23431 | 31556 | +34.7% |
| Cost | 0.0869 | 0.1399 | +61.0% | 0.1221 | 0.1609 | +31.8% |
| Cached vs Uncached Token Ratio | 10.25:1 | 10.43:1 | +1.8% | 6.89:1 | 7.76:1 | +12.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_batch (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 37.63s | 136.66s | +263.1% | 57.70s | 67.38s | +16.8% |
| Process Start → First Event | 0.74s | 0.74s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.75s | 5.77s | — | 4.97s | 7.61s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 4 | 17 | +325.0% | 8 | 9 | +12.5% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 4 | 17 | +325.0% | 8 | 9 | +12.5% |
| Output Tokens | 1245 | 3153 | +153.3% | 1653 | 1423 | -13.9% |
| Reasoning / Thinking Tokens | 392 | 1809 | +361.5% | 477 | 440 | -7.8% |
| Total Input Tokens | 99434 | 542656 | +445.7% | 182990 | 372069 | +103.3% |
| Cached Input Tokens | 88832 | 508672 | +472.6% | 167680 | 337152 | +101.1% |
| Uncached Input Tokens | 10602 | 33984 | +220.5% | 15310 | 34917 | +128.1% |
| Cost | 0.0692 | 0.2742 | +296.3% | 0.1068 | 0.1949 | +82.4% |
| Cached vs Uncached Token Ratio | 8.38:1 | 14.97:1 | +78.6% | 10.95:1 | 9.66:1 | -11.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): read Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515679910409000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515679910409000/audit: is a directory |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | ? ✗ |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | ? ✗ |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 94.77s | 88.63s | -6.5% | 34.78s | 71.88s | +106.7% |
| Process Start → First Event | 0.14s | 0.12s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 6.02s | 6.20s | — | 4.62s | 9.21s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 11 | 16 | +45.5% | 4 | 10 | +150.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | 16 | +45.5% | 4 | 10 | +150.0% |
| Output Tokens | 1625 | 1891 | +16.4% | 906 | 1271 | +40.3% |
| Reasoning / Thinking Tokens | 423 | 512 | +21.0% | 273 | 393 | +44.0% |
| Total Input Tokens | 144065 | 349042 | +142.3% | 120508 | 250782 | +108.1% |
| Cached Input Tokens | 122112 | 323328 | +164.8% | 108032 | 231680 | +114.5% |
| Uncached Input Tokens | 21953 | 25714 | +17.1% | 12476 | 19102 | +53.1% |
| Cost | 0.1110 | 0.1752 | +57.8% | 0.0729 | 0.1265 | +73.4% |
| Cached vs Uncached Token Ratio | 5.56:1 | 12.57:1 | +126.1% | 8.66:1 | 12.13:1 | +40.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ? ✗ | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516276227048000/service/dispatch.go: symbol not found |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ? ✗ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ? ✗ |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
default prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 46.04s | 72.36s | +57.2% | 38.95s | 52.80s | +35.5% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.14s | — |
| First Event → First Tool Call | 5.81s | 6.16s | — | 4.84s | 4.89s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 13 | +160.0% | 7 | 9 | +28.6% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 13 | +160.0% | 7 | 9 | +28.6% |
| Output Tokens | 1406 | 1324 | -5.8% | 1175 | 1311 | +11.6% |
| Reasoning / Thinking Tokens | 356 | 422 | +18.5% | 320 | 390 | +21.9% |
| Total Input Tokens | 120118 | 237330 | +97.6% | 183988 | 291276 | +58.3% |
| Cached Input Tokens | 107008 | 198912 | +85.9% | 158464 | 264448 | +66.9% |
| Uncached Input Tokens | 13110 | 38418 | +193.0% | 25524 | 26828 | +5.1% |
| Cost | 0.0816 | 0.1676 | +105.5% | 0.1221 | 0.1544 | +26.5% |
| Cached vs Uncached Token Ratio | 8.16:1 | 5.18:1 | -36.6% | 6.21:1 | 9.86:1 | +58.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 6 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve probe: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ? ✗ |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_batch (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 41.40s | 85.04s | +105.4% | 47.65s | 50.42s | +5.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.85s | 6.16s | — | 5.49s | 7.28s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 13 | +116.7% | 6 | 11 | +83.3% |
| Initial Load / Discovery Turns | 1 | 0 | -100.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 13 | +116.7% | 6 | 11 | +83.3% |
| Output Tokens | 1376 | 1637 | +19.0% | 1283 | 946 | -26.3% |
| Reasoning / Thinking Tokens | 448 | 441 | -1.6% | 376 | 0 | -100.0% |
| Total Input Tokens | 162668 | 318322 | +95.7% | 139643 | 260479 | +86.5% |
| Cached Input Tokens | 147456 | 292608 | +98.4% | 127232 | 238848 | +87.7% |
| Uncached Input Tokens | 15212 | 25714 | +69.0% | 12411 | 21631 | +74.3% |
| Cost | 0.0977 | 0.1634 | +67.3% | 0.0836 | 0.1256 | +50.3% |
| Cached vs Uncached Token Ratio | 9.69:1 | 11.38:1 | +17.4% | 10.25:1 | 11.04:1 | +7.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_organize_imports (✓ ✓) |
| 11 | — | ? ✗ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 53.51s | 77.14s | +44.2% | 52.56s | 112.20s | +113.5% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 5.52s | 5.67s | — | 5.56s | 6.13s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 10 | 16 | +60.0% | 6 | 16 | +166.7% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 10 | 16 | +60.0% | 6 | 16 | +166.7% |
| Output Tokens | 1637 | 1483 | -9.4% | 1329 | 1881 | +41.5% |
| Reasoning / Thinking Tokens | 509 | 502 | -1.4% | 507 | 900 | +77.5% |
| Total Input Tokens | 185359 | 251647 | +35.8% | 165115 | 357981 | +116.8% |
| Cached Input Tokens | 170752 | 221184 | +29.5% | 150528 | 321024 | +113.3% |
| Uncached Input Tokens | 14607 | 30463 | +108.6% | 14587 | 36957 | +153.4% |
| Cost | 0.1060 | 0.1563 | +47.4% | 0.0970 | 0.2074 | +113.7% |
| Cached vs Uncached Token Ratio | 11.69:1 | 7.26:1 | -37.9% | 10.32:1 | 8.69:1 | -15.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516147968342000/audit/normalize.go: symbol not found |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_replace_body", "symbol": "Classify", "status": "ok", "diff": "--- a/Classify\n+++ b/Classify\n-switch kind {\n-\tcase \"health\":\n-\t\treturn \"control\"\n-\tdefault:\n-\t\treturn \"data\"\n-\t}\n+switch NormalizeKind(kind) {\n+case \"health\", \"probe\":\n+\treturn \"control\"\n+default:\n+\treturn \"data\"\n+}\n" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch |
| 9 | — | semedit/semantic_outline (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 59.58s | 172.98s | +190.3% | 30.00s | 88.28s | +194.2% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 7.52s | 21.43s | — | 4.86s | 5.38s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 14 | +100.0% | 6 | 13 | +116.7% |
| Initial Load / Discovery Turns | 1 | 3 | +200.0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 14 | +100.0% | 6 | 13 | +116.7% |
| Output Tokens | 1826 | 1798 | -1.5% | 768 | 1697 | +121.0% |
| Reasoning / Thinking Tokens | 325 | 461 | +41.8% | 0 | 708 | — |
| Total Input Tokens | 165073 | 236915 | +43.5% | 160328 | 297031 | +85.3% |
| Cached Input Tokens | 149504 | 215552 | +44.2% | 146432 | 265728 | +81.5% |
| Uncached Input Tokens | 15569 | 21363 | +37.2% | 13896 | 31303 | +125.3% |
| Cost | 0.1032 | 0.1355 | +31.3% | 0.0809 | 0.1748 | +115.9% |
| Cached vs Uncached Token Ratio | 9.60:1 | 10.09:1 | +5.1% | 10.54:1 | 8.49:1 | -19.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | ? ✗ |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "symbol": "Classify", "status": "ok", "diff": "--- a/Classify\n+++ b/Classify\n-switch kind {\n-\tcase \"health\":\n-\t\treturn \"control\"\n-\tdefault:\n-\t\treturn \"data\"\n-\t}\n+switch NormalizeKind(kind) {\n+case \"health\", \"probe\":\n+\treturn \"control\"\n+default:\n+\treturn \"data\"\n+}\n" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Reco |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_organize_imports (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 44.94s | 115.96s | +158.0% | 37.02s | 84.68s | +128.7% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 6.17s | 5.69s | — | 6.73s | 6.55s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 14 | +100.0% | 6 | 13 | +116.7% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 14 | +100.0% | 6 | 13 | +116.7% |
| Output Tokens | 1270 | 2243 | +76.6% | 749 | 1462 | +95.2% |
| Reasoning / Thinking Tokens | 202 | 1166 | +477.2% | 0 | 489 | — |
| Total Input Tokens | 160290 | 346520 | +116.2% | 160229 | 313795 | +95.8% |
| Cached Input Tokens | 146432 | 302592 | +106.6% | 147456 | 280320 | +90.1% |
| Uncached Input Tokens | 13858 | 43928 | +217.0% | 12773 | 33475 | +162.1% |
| Cost | 0.0897 | 0.2281 | +154.4% | 0.0782 | 0.1782 | +127.9% |
| Cached vs Uncached Token Ratio | 10.57:1 | 6.89:1 | -34.8% | 11.54:1 | 8.37:1 | -27.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516251026513000/audit/event.go: symbol not found |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518447825877000/audit/normalize.go: symbol not found |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_batch (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 53.23s | 73.24s | +37.6% | 56.65s | 103.99s | +83.6% |
| Process Start → First Event | 0.79s | 0.12s | — | 0.14s | 0.52s | — |
| First Event → First Tool Call | 6.51s | 7.43s | — | 5.26s | 6.15s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 13 | +85.7% | 7 | 17 | +142.9% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 13 | +85.7% | 7 | 17 | +142.9% |
| Output Tokens | 1399 | 1205 | -13.9% | 1839 | 2045 | +11.2% |
| Reasoning / Thinking Tokens | 315 | 310 | -1.6% | 488 | 818 | +67.6% |
| Total Input Tokens | 161006 | 393452 | +144.4% | 167255 | 415861 | +148.6% |
| Cached Input Tokens | 146432 | 350720 | +139.5% | 152576 | 380928 | +149.7% |
| Uncached Input Tokens | 14574 | 42732 | +193.2% | 14679 | 34933 | +138.0% |
| Cost | 0.0945 | 0.2134 | +125.9% | 0.1039 | 0.2184 | +110.1% |
| Cached vs Uncached Token Ratio | 10.05:1 | 8.21:1 | -18.3% | 10.39:1 | 10.90:1 | +4.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516073348707000/audit/event.go: symbol not found |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516073348707000/audit/normalize.go: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | ? ✗ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518227244160000/service/dispatch.go: symbol not found |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ? ✗ |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 61.82s | 69.34s | +12.2% | 40.69s | 89.45s | +119.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 9.34s | 5.74s | — | 4.58s | 5.57s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 14 | +75.0% | 7 | 15 | +114.3% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 14 | +75.0% | 7 | 15 | +114.3% |
| Output Tokens | 1763 | 1282 | -27.3% | 1226 | 1611 | +31.4% |
| Reasoning / Thinking Tokens | 410 | 389 | -5.1% | 231 | 397 | +71.9% |
| Total Input Tokens | 203583 | 251885 | +23.7% | 162108 | 361233 | +122.8% |
| Cached Input Tokens | 186880 | 228864 | +22.5% | 147456 | 328704 | +122.9% |
| Uncached Input Tokens | 16703 | 23021 | +37.8% | 14652 | 32529 | +122.0% |
| Cost | 0.1156 | 0.1357 | +17.3% | 0.0917 | 0.1886 | +105.7% |
| Cached vs Uncached Token Ratio | 11.19:1 | 9.94:1 | -11.1% | 10.06:1 | 10.10:1 | +0.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve health: symbol not found |
| 5 | ? ✗ | semedit/semantic_lookup (✓ ✗): resolve probe: symbol not found |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_batch (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | ? ✗ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 38.26s | 89.79s | +134.7% | 37.45s | 66.89s | +78.6% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 5.67s | 5.34s | — | 5.94s | 8.65s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 15 | +150.0% | 5 | 12 | +140.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 15 | +150.0% | 5 | 12 | +140.0% |
| Output Tokens | 1038 | 1409 | +35.7% | 775 | 1302 | +68.0% |
| Reasoning / Thinking Tokens | 227 | 420 | +85.0% | 0 | 311 | — |
| Total Input Tokens | 161070 | 261416 | +62.3% | 141922 | 329178 | +131.9% |
| Cached Input Tokens | 147456 | 229120 | +55.4% | 120064 | 297472 | +147.8% |
| Uncached Input Tokens | 13614 | 32296 | +137.2% | 21858 | 31706 | +45.1% |
| Cost | 0.0867 | 0.1609 | +85.5% | 0.0943 | 0.1738 | +84.2% |
| Cached vs Uncached Token Ratio | 10.83:1 | 7.09:1 | -34.5% | 5.49:1 | 9.38:1 | +70.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515780768791000/audit/normalize.go: symbol not found |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_batch (✓ ✓) |
| 7 | — | ? ✗ |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 41.63s | 91.82s | +120.6% | 43.20s | 64.14s | +48.5% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.70s | 6.04s | — | 5.43s | 9.13s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 21 | +200.0% | 8 | 10 | +25.0% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 21 | +200.0% | 8 | 10 | +25.0% |
| Output Tokens | 1308 | 1657 | +26.7% | 1318 | 1110 | -15.8% |
| Reasoning / Thinking Tokens | 448 | 521 | +16.3% | 174 | 390 | +124.1% |
| Total Input Tokens | 180722 | 288411 | +59.6% | 184593 | 230241 | +24.7% |
| Cached Input Tokens | 165632 | 257792 | +55.6% | 169728 | 198912 | +17.2% |
| Uncached Input Tokens | 15090 | 30619 | +102.9% | 14865 | 31329 | +110.8% |
| Cost | 0.1011 | 0.1682 | +66.4% | 0.0982 | 0.1468 | +49.4% |
| Cached vs Uncached Token Ratio | 10.98:1 | 8.42:1 | -23.3% | 11.42:1 | 6.35:1 | -44.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | ? ✗ |
| 10 | — | semedit/semantic_scaffold_file (✓ ✗): file already exists: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515977150161000/audit/normalize.go |
| 11 | — | semedit/semantic_insert_function (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | ? ✗ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_verify (✓ ✓) |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 54.12s | 100.78s | +86.2% | 33.56s | 77.23s | +130.1% |
| Process Start → First Event | 0.70s | 0.82s | — | 0.12s | 0.16s | — |
| First Event → First Tool Call | 6.65s | 4.41s | — | 6.09s | 6.27s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 18 | +157.1% | 5 | 16 | +220.0% |
| Initial Load / Discovery Turns | 0 | 1 | — | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 18 | +157.1% | 5 | 16 | +220.0% |
| Output Tokens | 1099 | 2456 | +123.5% | 798 | 1397 | +75.1% |
| Reasoning / Thinking Tokens | 325 | 1081 | +232.6% | 146 | 445 | +204.8% |
| Total Input Tokens | 201349 | 563911 | +180.1% | 141534 | 338475 | +139.1% |
| Cached Input Tokens | 185856 | 518144 | +178.8% | 128256 | 305664 | +138.3% |
| Uncached Input Tokens | 15493 | 45767 | +195.4% | 13278 | 32811 | +147.1% |
| Cost | 0.1030 | 0.2882 | +179.8% | 0.0771 | 0.1815 | +135.5% |
| Cached vs Uncached Token Ratio | 12.00:1 | 11.32:1 | -5.6% | 9.66:1 | 9.32:1 | -3.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515679910424000/audit/normalize.go: symbol not found |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790515679910424000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned by the delivery b |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_organize_imports (✓ ✓) |
| 13 | — | ? ✗ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ? ✗ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): read Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790517865676480000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790517865676480000/audit: is a directory |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ? ✗ |
| 15 | — | ? ✗ |
| 16 | — | ✓ ✓ |
default prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 32.24s | 65.51s | +103.2% | 45.09s | 82.53s | +83.1% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.87s | 5.56s | — | 6.06s | 6.12s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 12 | +140.0% | 7 | 14 | +100.0% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 0 | 1 | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 12 | +140.0% | 7 | 14 | +100.0% |
| Output Tokens | 828 | 837 | +1.1% | 1253 | 1495 | +19.3% |
| Reasoning / Thinking Tokens | 186 | 0 | -100.0% | 243 | 601 | +147.3% |
| Total Input Tokens | 139761 | 357002 | +155.4% | 162866 | 238787 | +46.6% |
| Cached Input Tokens | 127232 | 327680 | +157.5% | 148480 | 205056 | +38.1% |
| Uncached Input Tokens | 12529 | 29322 | +134.0% | 14386 | 33731 | +134.5% |
| Cost | 0.0758 | 0.1657 | +118.6% | 0.0918 | 0.1618 | +76.3% |
| Cached vs Uncached Token Ratio | 10.16:1 | 11.18:1 | +10.0% | 10.32:1 | 6.08:1 | -41.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Dispatcher.Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516370218693000/audit/sink.go: symbol not found |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ? ✗ |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_batch (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 44.68s | 67.36s | +50.8% | 43.14s | 118.99s | +175.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.54s | 0.12s | — |
| First Event → First Tool Call | 9.25s | 4.82s | — | 7.21s | 5.68s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 10 | +66.7% | 9 | 15 | +66.7% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 0 | -100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 10 | +66.7% | 9 | 15 | +66.7% |
| Output Tokens | 1045 | 1133 | +8.4% | 997 | 2570 | +157.8% |
| Reasoning / Thinking Tokens | 286 | 310 | +8.4% | 146 | 1327 | +808.9% |
| Total Input Tokens | 161937 | 238454 | +47.3% | 204825 | 466527 | +127.8% |
| Cached Input Tokens | 127232 | 217600 | +71.0% | 188928 | 429568 | +127.4% |
| Uncached Input Tokens | 34705 | 20854 | -39.9% | 15897 | 36959 | +132.5% |
| Cost | 0.1352 | 0.1246 | -7.9% | 0.1013 | 0.2485 | +145.4% |
| Cached vs Uncached Token Ratio | 3.67:1 | 10.43:1 | +184.6% | 11.88:1 | 11.62:1 | -2.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 3 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | ? ✗ |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518507756063000/service/dispatch.go: symbol not found |
| 10 | — | ? ✗ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.28s | 100.23s | +107.6% | 41.96s | 77.31s | +84.2% |
| Process Start → First Event | 0.12s | 0.14s | — | 0.12s | 0.20s | — |
| First Event → First Tool Call | 7.99s | 7.32s | — | 6.93s | 11.69s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 20 | +233.3% | 7 | 13 | +85.7% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 0 | -100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 20 | +233.3% | 7 | 13 | +85.7% |
| Output Tokens | 1462 | 1787 | +22.2% | 1026 | 1727 | +68.3% |
| Reasoning / Thinking Tokens | 404 | 497 | +23.0% | 282 | 910 | +222.7% |
| Total Input Tokens | 141831 | 402807 | +184.0% | 183658 | 222362 | +21.1% |
| Cached Input Tokens | 127232 | 368640 | +189.7% | 168704 | 192768 | +14.3% |
| Uncached Input Tokens | 14599 | 34167 | +134.0% | 14954 | 29594 | +97.9% |
| Cost | 0.0916 | 0.2061 | +125.0% | 0.0959 | 0.1551 | +61.8% |
| Cached vs Uncached Token Ratio | 8.72:1 | 10.79:1 | +23.8% | 11.28:1 | 6.51:1 | -42.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ? ✗ | ✓ ✓ |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_scaffold_file (✓ ✗): file already exists: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516523962131000/audit/normalize.go |
| 10 | — | ? ✗ |
| 11 | — | semedit/semantic_insert_function (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_batch (✓ ✓) |
| 14 | — | ? ✗ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide verification may format and mutate test files, conflicting with the explicit prohibition on editing tests; use a read-only or narrowly scoped check instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that re |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 52.40s | 78.41s | +49.6% | 49.70s | 62.40s | +25.6% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.14s | 0.14s | — |
| First Event → First Tool Call | 5.83s | 7.62s | — | 5.83s | 5.96s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 13 | +116.7% | 7 | 13 | +85.7% |
| Initial Load / Discovery Turns | 1 | 3 | +200.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 13 | +116.7% | 7 | 13 | +85.7% |
| Output Tokens | 1589 | 1453 | -8.6% | 1603 | 1370 | -14.5% |
| Reasoning / Thinking Tokens | 437 | 572 | +30.9% | 271 | 453 | +67.2% |
| Total Input Tokens | 142245 | 362505 | +154.8% | 204595 | 242477 | +18.5% |
| Cached Input Tokens | 109056 | 331776 | +204.2% | 188928 | 209152 | +10.7% |
| Uncached Input Tokens | 33189 | 30729 | -7.4% | 15667 | 33325 | +112.7% |
| Cost | 0.1356 | 0.1851 | +36.5% | 0.1098 | 0.1584 | +44.2% |
| Cached vs Uncached Token Ratio | 3.29:1 | 10.80:1 | +228.6% | 12.06:1 | 6.28:1 | -48.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 47.43s | 56.90s | +20.0% | 39.29s | 87.20s | +121.9% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.59s | 4.59s | — | 4.56s | 5.19s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 9 | 10 | +11.1% | 7 | 13 | +85.7% |
| Initial Load / Discovery Turns | 2 | 3 | +50.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 9 | 10 | +11.1% | 7 | 13 | +85.7% |
| Output Tokens | 1528 | 1188 | -22.3% | 1235 | 1634 | +32.3% |
| Reasoning / Thinking Tokens | 306 | 462 | +51.0% | 237 | 694 | +192.8% |
| Total Input Tokens | 205189 | 234054 | +14.1% | 162416 | 424237 | +161.2% |
| Cached Input Tokens | 188928 | 214528 | +13.6% | 148480 | 387840 | +161.2% |
| Uncached Input Tokens | 16261 | 19526 | +20.1% | 13936 | 36397 | +161.2% |
| Cost | 0.1108 | 0.1231 | +11.1% | 0.0904 | 0.2171 | +140.2% |
| Cached vs Uncached Token Ratio | 11.62:1 | 10.99:1 | -5.4% | 10.65:1 | 10.66:1 | +0.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ? ✗ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 39.14s | 70.75s | +80.8% | 51.05s | 92.18s | +80.6% |
| Process Start → First Event | 0.11s | 0.21s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 4.86s | 9.52s | — | 7.05s | 6.76s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 15 | +150.0% | 7 | 15 | +114.3% |
| Initial Load / Discovery Turns | 3 | 0 | -100.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 15 | +150.0% | 7 | 15 | +114.3% |
| Output Tokens | 1193 | 1182 | -0.9% | 1596 | 1670 | +4.6% |
| Reasoning / Thinking Tokens | 214 | 383 | +79.0% | 284 | 562 | +97.9% |
| Total Input Tokens | 138820 | 250600 | +80.5% | 162336 | 408611 | +151.7% |
| Cached Input Tokens | 119040 | 217088 | +82.4% | 148480 | 374784 | +152.4% |
| Uncached Input Tokens | 19780 | 33512 | +69.4% | 13856 | 33827 | +144.1% |
| Cost | 0.0968 | 0.1576 | +62.8% | 0.0953 | 0.2062 | +116.4% |
| Cached vs Uncached Token Ratio | 6.02:1 | 6.48:1 | +7.6% | 10.72:1 | 11.08:1 | +3.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | ? ✗ |
| 7 | — | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ? ✗ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518016020952000/audit/normalize.go: symbol not found |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_scaffold_file (✓ ✗): file already exists: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518016020952000/audit/normalize.go |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_batch (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 47.70s | 63.96s | +34.1% | 35.49s | 95.79s | +169.9% |
| Process Start → First Event | 0.13s | 0.47s | — | 0.14s | 0.13s | — |
| First Event → First Tool Call | 11.06s | 7.52s | — | 4.77s | 7.20s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 14 | +180.0% | 6 | 15 | +150.0% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 14 | +180.0% | 6 | 15 | +150.0% |
| Output Tokens | 1390 | 1237 | -11.0% | 1035 | 1498 | +44.7% |
| Reasoning / Thinking Tokens | 323 | 469 | +45.2% | 144 | 509 | +253.5% |
| Total Input Tokens | 142430 | 224352 | +57.5% | 140727 | 312334 | +121.9% |
| Cached Input Tokens | 129280 | 186624 | +44.4% | 128256 | 260096 | +102.8% |
| Uncached Input Tokens | 13150 | 37728 | +186.9% | 12471 | 52238 | +318.9% |
| Cost | 0.0866 | 0.1623 | +87.4% | 0.0780 | 0.2207 | +183.0% |
| Cached vs Uncached Token Ratio | 9.83:1 | 4.95:1 | -49.7% | 10.28:1 | 4.98:1 | -51.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ? ✗ | ? ✗ |
| 6 | — | ✓ ✓ |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_batch (✓ ✓) |
| 10 | — | semedit/semantic_organize_imports (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ? ✗ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 32.89s | 69.16s | +110.2% | 52.49s | 110.74s | +111.0% |
| Process Start → First Event | 0.14s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 5.55s | 5.97s | — | 5.09s | 6.22s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 14 | +133.3% | 8 | 21 | +162.5% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 3 | +200.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 14 | +133.3% | 8 | 21 | +162.5% |
| Output Tokens | 848 | 1411 | +66.4% | 1561 | 1841 | +17.9% |
| Reasoning / Thinking Tokens | 0 | 504 | — | 388 | 767 | +97.7% |
| Total Input Tokens | 140720 | 345962 | +145.9% | 183461 | 378737 | +106.4% |
| Cached Input Tokens | 128256 | 305664 | +138.3% | 158464 | 344320 | +117.3% |
| Uncached Input Tokens | 12464 | 40298 | +223.3% | 24997 | 34417 | +37.7% |
| Cost | 0.0738 | 0.2011 | +172.4% | 0.1265 | 0.2047 | +61.9% |
| Cached vs Uncached Token Ratio | 10.29:1 | 7.59:1 | -26.3% | 6.34:1 | 10.00:1 | +57.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ? ✗ |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide Go verification may format and write files, potentially modifying tests despite the user's explicit prohibition; a narrower source-only verification is safer. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Go verification formats and may write the entire audit directory, including tests, conflicting with the explicit instruction not to edit tests; verify only the changed non-test files or use a read-only check. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for |
| 14 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): read Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518196393630000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518196393630000/audit: is a directory |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518196393630000/audit/normalize.go: symbol not found |
| 11 | — | semedit/semantic_outline (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_insert_function (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518196393630000/service/dispatch.go: symbol not found |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
| 21 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 34.63s | 67.82s | +95.8% | 27.78s | 72.17s | +159.8% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.99s | 7.47s | — | 4.65s | 6.72s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 17 | +183.3% | 6 | 13 | +116.7% |
| Initial Load / Discovery Turns | 2 | 4 | +100.0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 17 | +183.3% | 6 | 13 | +116.7% |
| Output Tokens | 1032 | 972 | -5.8% | 689 | 1027 | +49.1% |
| Reasoning / Thinking Tokens | 290 | 123 | -57.6% | 0 | 83 | — |
| Total Input Tokens | 158968 | 290037 | +82.4% | 158348 | 410415 | +159.2% |
| Cached Input Tokens | 125952 | 259840 | +106.3% | 144384 | 385280 | +166.8% |
| Uncached Input Tokens | 33016 | 30197 | -8.5% | 13964 | 25135 | +80.0% |
| Cost | 0.1306 | 0.1541 | +18.1% | 0.0796 | 0.1730 | +117.3% |
| Cached vs Uncached Token Ratio | 3.81:1 | 8.60:1 | +125.6% | 10.34:1 | 15.33:1 | +48.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Dispatcher.Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516128797211000/audit/event.go: symbol not found |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_lookup (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ? ✗ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ? ✗ |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 47.55s | 102.84s | +116.3% | 51.83s | 100.98s | +94.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 6.26s | 6.26s | — | 4.73s | 5.44s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 20 | +185.7% | 7 | 17 | +142.9% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 20 | +185.7% | 7 | 17 | +142.9% |
| Output Tokens | 1282 | 2405 | +87.6% | 1516 | 2066 | +36.3% |
| Reasoning / Thinking Tokens | 404 | 871 | +115.6% | 357 | 779 | +118.2% |
| Total Input Tokens | 182130 | 371966 | +104.2% | 162157 | 419235 | +158.5% |
| Cached Input Tokens | 167680 | 339200 | +102.3% | 146432 | 387584 | +164.7% |
| Uncached Input Tokens | 14450 | 32766 | +126.8% | 15725 | 31651 | +101.3% |
| Cost | 0.0991 | 0.2077 | +109.5% | 0.0993 | 0.2116 | +113.0% |
| Cached vs Uncached Token Ratio | 11.60:1 | 10.35:1 | -10.8% | 9.31:1 | 12.25:1 | +31.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516421045861000/audit/normalize.go: symbol not found |
| 5 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_scaffold_file (✓ ✗): file already exists: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516421045861000/audit/normalize.go |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790516421045861000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned by the delivery b |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_organize_imports (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve IsControlTraffic: symbol not found |
| 7 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790518625734207000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned by the delivery b |
| 12 | — | semedit/semantic_organize_imports (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 45.29s | — | — | 44.78s | 69.47s | +55.1% |
| Process Start → First Event | 0.12s | — | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 5.09s | — | — | 5.34s | 7.08s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | — | — | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | — | — | 7 | 14 | +100.0% |
| Initial Load / Discovery Turns | 1 | — | — | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | — | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | — | — | 7 | 14 | +100.0% |
| Output Tokens | 1348 | — | — | 1293 | 1305 | +0.9% |
| Reasoning / Thinking Tokens | 537 | — | — | 480 | 310 | -35.4% |
| Total Input Tokens | 140433 | — | — | 183749 | 341663 | +85.9% |
| Cached Input Tokens | 127232 | — | — | 169728 | 315392 | +85.8% |
| Uncached Input Tokens | 13201 | — | — | 14021 | 26271 | +87.4% |
| Cost | 0.0884 | — | — | 0.0996 | 0.1647 | +65.3% |
| Cached vs Uncached Token Ratio | 9.64:1 | — | — | 12.11:1 | 12.01:1 | -0.8% |
| Oracle L1: Mutation Policy | PASS | — | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | — | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | — | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | — | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | — | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | not published |
| 2 | ✓ ✓ | not published |
| 3 | ? ✗ | not published |
| 4 | ✓ ✓ | not published |
| 5 | ✓ ✓ | not published |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.91s | 85.61s | +75.1% | 30.41s | 110.27s | +262.7% |
| Process Start → First Event | 0.14s | 0.14s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 10.65s | 13.39s | — | 5.69s | 7.50s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 16 | +166.7% | 4 | 15 | +275.0% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 16 | +166.7% | 4 | 15 | +275.0% |
| Output Tokens | 1291 | 1986 | +53.8% | 847 | 2037 | +140.5% |
| Reasoning / Thinking Tokens | 404 | 868 | +114.9% | 223 | 682 | +205.8% |
| Total Input Tokens | 224204 | 461004 | +105.6% | 139392 | 431254 | +209.4% |
| Cached Input Tokens | 200704 | 425728 | +112.1% | 123392 | 401408 | +225.3% |
| Uncached Input Tokens | 23500 | 35276 | +50.1% | 16000 | 29846 | +86.5% |
| Cost | 0.1301 | 0.2303 | +77.0% | 0.0842 | 0.2090 | +148.1% |
| Cached vs Uncached Token Ratio | 8.54:1 | 12.07:1 | +41.3% | 7.71:1 | 13.45:1 | +74.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | — | semedit/semantic_lookup (✓ ✓) |
| 6 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790518886911695000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned b |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: semantic_verify on `.` may format and modify repository-wide Go files, including protected tests, contrary to the user's explicit no-test-edits constraint; verify only the changed source files instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirm |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 46.02s | 88.22s | +91.7% | 49.78s | 120.14s | +141.4% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 8.57s | 8.29s | — | 5.09s | 9.94s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 14 | +180.0% | 6 | 17 | +183.3% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 3 | +200.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 14 | +180.0% | 6 | 17 | +183.3% |
| Output Tokens | 1632 | 1959 | +20.0% | 1567 | 2506 | +59.9% |
| Reasoning / Thinking Tokens | 728 | 863 | +18.5% | 374 | 974 | +160.4% |
| Total Input Tokens | 172877 | 513645 | +197.1% | 149027 | 470042 | +215.4% |
| Cached Input Tokens | 156672 | 479232 | +205.9% | 133376 | 428032 | +220.9% |
| Uncached Input Tokens | 16205 | 34413 | +112.4% | 15651 | 42010 | +168.4% |
| Cost | 0.1092 | 0.2411 | +120.8% | 0.0967 | 0.2555 | +164.2% |
| Cached vs Uncached Token Ratio | 9.67:1 | 13.93:1 | +44.0% | 8.52:1 | 10.19:1 | +19.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517102958664000/service/dispatch.go: symbol not found |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: semantic_verify on the entire workspace can format and write Go files, including protected tests, while the user explicitly forbade test edits. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why i |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 70.70s | 102.81s | +45.4% | 54.97s | 115.33s | +109.8% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 12.14s | 12.97s | — | 6.41s | 6.43s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 13 | +85.7% | 7 | 19 | +171.4% |
| Initial Load / Discovery Turns | 1 | 0 | -100.0% | 1 | 3 | +200.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 13 | +85.7% | 7 | 19 | +171.4% |
| Output Tokens | 1953 | 1909 | -2.3% | 1670 | 1923 | +15.1% |
| Reasoning / Thinking Tokens | 1018 | 930 | -8.6% | 388 | 773 | +99.2% |
| Total Input Tokens | 321281 | 402305 | +25.2% | 169173 | 419380 | +147.9% |
| Cached Input Tokens | 294144 | 371456 | +26.3% | 153600 | 378112 | +146.2% |
| Uncached Input Tokens | 27137 | 30849 | +13.7% | 15573 | 41268 | +165.0% |
| Cost | 0.1785 | 0.2055 | +15.1% | 0.1031 | 0.2314 | +124.5% |
| Cached vs Uncached Token Ratio | 10.84:1 | 12.04:1 | +11.1% | 9.86:1 | 9.16:1 | -7.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 50.16s | 65.57s | +30.7% | 53.56s | 102.01s | +90.5% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 10.50s | 12.25s | — | 6.36s | 13.45s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 4 | 8 | +100.0% | 6 | 11 | +83.3% |
| Initial Load / Discovery Turns | 1 | 3 | +200.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 4 | 8 | +100.0% | 6 | 11 | +83.3% |
| Output Tokens | 1960 | 1552 | -20.8% | 1897 | 2248 | +18.5% |
| Reasoning / Thinking Tokens | 764 | 517 | -32.3% | 575 | 1030 | +79.1% |
| Total Input Tokens | 196257 | 292976 | +49.3% | 228474 | 466598 | +104.2% |
| Cached Input Tokens | 174336 | 258816 | +48.5% | 203776 | 432640 | +112.3% |
| Uncached Input Tokens | 21921 | 34160 | +55.8% | 24698 | 33958 | +37.5% |
| Cost | 0.1324 | 0.1760 | +32.9% | 0.1436 | 0.2340 | +63.0% |
| Cached vs Uncached Token Ratio | 7.95:1 | 7.58:1 | -4.7% | 8.25:1 | 12.74:1 | +54.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_batch (✓ ✓) |
| 8 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_batch (✓ ✓) |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 56.74s | 94.62s | +66.7% | 51.35s | 64.91s | +26.4% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 8.17s | 11.90s | — | 5.06s | 8.06s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 13 | +85.7% | 7 | 11 | +57.1% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 13 | +85.7% | 7 | 11 | +57.1% |
| Output Tokens | 2117 | 1935 | -8.6% | 1849 | 1484 | -19.7% |
| Reasoning / Thinking Tokens | 738 | 694 | -6.0% | 490 | 596 | +21.6% |
| Total Input Tokens | 213741 | 380168 | +77.9% | 259457 | 365001 | +40.7% |
| Cached Input Tokens | 186880 | 345344 | +84.8% | 234240 | 328192 | +40.1% |
| Uncached Input Tokens | 26861 | 34824 | +29.6% | 25217 | 36809 | +46.0% |
| Cost | 0.1496 | 0.2063 | +37.9% | 0.1508 | 0.2001 | +32.6% |
| Cached vs Uncached Token Ratio | 6.96:1 | 9.92:1 | +42.5% | 9.29:1 | 8.92:1 | -4.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 115.90s | 129.69s | +11.9% | 46.25s | 118.69s | +156.6% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.15s | — |
| First Event → First Tool Call | 61.17s | 18.72s | — | 9.56s | 9.49s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 19 | +280.0% | 7 | 12 | +71.4% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 0 | 1 | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 19 | +280.0% | 7 | 12 | +71.4% |
| Output Tokens | 1422 | 2388 | +67.9% | 1533 | 2540 | +65.7% |
| Reasoning / Thinking Tokens | 655 | 1090 | +66.4% | 508 | 1508 | +196.9% |
| Total Input Tokens | 286005 | 396392 | +38.6% | 257964 | 314366 | +21.9% |
| Cached Input Tokens | 269824 | 351744 | +30.4% | 232192 | 285184 | +22.8% |
| Uncached Input Tokens | 16181 | 44648 | +175.9% | 25772 | 29182 | +13.2% |
| Cost | 0.1339 | 0.2430 | +81.5% | 0.1480 | 0.1949 | +31.7% |
| Cached vs Uncached Token Ratio | 16.68:1 | 7.88:1 | -52.8% | 9.01:1 | 9.77:1 | +8.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517336539018000/audit/normalize.go: symbol not found |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Dispatcher.Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517336539018000/audit/sink.go: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_batch (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve probe: symbol not found |
| 5 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.63s | 69.94s | +43.8% | 43.35s | 118.43s | +173.2% |
| Process Start → First Event | 0.12s | 0.11s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 11.00s | 10.19s | — | 7.68s | 9.13s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 9 | +80.0% | 6 | 17 | +183.3% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 9 | +80.0% | 6 | 17 | +183.3% |
| Output Tokens | 1516 | 1567 | +3.4% | 1278 | 2254 | +76.4% |
| Reasoning / Thinking Tokens | 503 | 500 | -0.6% | 550 | 857 | +55.8% |
| Total Input Tokens | 189654 | 358599 | +89.1% | 191343 | 463476 | +142.2% |
| Cached Input Tokens | 173824 | 328704 | +89.1% | 175872 | 423936 | +141.0% |
| Uncached Input Tokens | 15830 | 29895 | +88.9% | 15471 | 39540 | +155.6% |
| Cost | 0.1083 | 0.1828 | +68.8% | 0.1055 | 0.2437 | +131.0% |
| Cached vs Uncached Token Ratio | 10.98:1 | 11.00:1 | +0.1% | 11.37:1 | 10.72:1 | -5.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | ? ✗ |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | semedit/report_feedback (✓ ✓) |
| 11 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Verification may format and write across the entire workspace, potentially modifying tests or protected metadata despite the user’s explicit prohibition; a narrower read-only or changed-file verification is needed. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without ask |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 46.70s | 123.46s | +164.3% | 62.61s | 75.23s | +20.1% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 6.17s | 8.88s | — | 8.30s | 5.34s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 20 | +233.3% | 6 | 11 | +83.3% |
| Initial Load / Discovery Turns | 2 | 3 | +50.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 20 | +233.3% | 6 | 11 | +83.3% |
| Output Tokens | 1642 | 2554 | +55.5% | 2041 | 1948 | -4.6% |
| Reasoning / Thinking Tokens | 691 | 1190 | +72.2% | 681 | 885 | +30.0% |
| Total Input Tokens | 198335 | 606939 | +206.0% | 193156 | 349736 | +81.1% |
| Cached Input Tokens | 175360 | 563968 | +221.6% | 156672 | 319744 | +104.1% |
| Uncached Input Tokens | 22975 | 42971 | +87.0% | 36484 | 29992 | -17.8% |
| Cost | 0.1304 | 0.2952 | +126.3% | 0.1644 | 0.1903 | +15.8% |
| Cached vs Uncached Token Ratio | 7.63:1 | 13.12:1 | +72.0% | 4.29:1 | 10.66:1 | +148.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_outline (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_insert_function (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide Go verification may format and modify test files, conflicting with the user's explicit prohibition on editing tests; verify only the changed source files or use a read-only check. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: semantic_verify may format and write Go files under the entire audit path, potentially modifying protected tests despite the user's explicit prohibition; use a non-mutating verification scoped to production files instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work withou |
| 8 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: The verification may format and write all Go files under service, including protected tests, conflicting with the user's explicit no-test-edits instruction. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 43.13s | 81.63s | +89.3% | 49.62s | 80.51s | +62.2% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.15s | 0.14s | — |
| First Event → First Tool Call | 12.33s | 12.25s | — | 10.52s | 5.92s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 15 | +200.0% | 5 | 14 | +180.0% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 15 | +200.0% | 5 | 14 | +180.0% |
| Output Tokens | 1093 | 1886 | +72.6% | 1742 | 1821 | +4.5% |
| Reasoning / Thinking Tokens | 308 | 749 | +143.2% | 832 | 787 | -5.4% |
| Total Input Tokens | 257512 | 377738 | +46.7% | 225980 | 454872 | +101.3% |
| Cached Input Tokens | 234240 | 343296 | +46.6% | 202752 | 415744 | +105.1% |
| Uncached Input Tokens | 23272 | 34442 | +48.0% | 23228 | 39128 | +68.5% |
| Cost | 0.1343 | 0.2049 | +52.6% | 0.1409 | 0.2344 | +66.3% |
| Cached vs Uncached Token Ratio | 10.07:1 | 9.97:1 | -1.0% | 8.73:1 | 10.63:1 | +21.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | descriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 51.16s | 133.28s | +160.5% | 50.32s | 131.54s | +161.4% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.14s | 0.13s | — |
| First Event → First Tool Call | 8.44s | 8.19s | — | 11.95s | 14.07s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 17 | +112.5% | 6 | 18 | +200.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 0 | 2 | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 17 | +112.5% | 6 | 18 | +200.0% |
| Output Tokens | 1527 | 2153 | +41.0% | 1542 | 2574 | +66.9% |
| Reasoning / Thinking Tokens | 347 | 794 | +128.8% | 532 | 1017 | +91.2% |
| Total Input Tokens | 258977 | 365286 | +41.0% | 283944 | 529644 | +86.5% |
| Cached Input Tokens | 238080 | 327168 | +37.4% | 256512 | 486912 | +89.8% |
| Uncached Input Tokens | 20897 | 38118 | +82.4% | 27432 | 42732 | +55.8% |
| Cost | 0.1352 | 0.2139 | +58.2% | 0.1586 | 0.2734 | +72.4% |
| Cached vs Uncached Token Ratio | 11.39:1 | 8.58:1 | -24.7% | 9.35:1 | 11.39:1 | +21.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "ok" }, { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519792998770000/service/dispatch.go: symbol not found" } ], "final_diff": "--- a/audit/normalize.go\n+++ b/audit/normalize.go\n // Package audit leaves room for normalization rules owned b |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_organize_imports (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 54.34s | 138.45s | +154.8% | 50.45s | 87.84s | +74.1% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 11.27s | 10.06s | — | 11.36s | 7.93s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 17 | +112.5% | 6 | 12 | +100.0% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 17 | +112.5% | 6 | 12 | +100.0% |
| Output Tokens | 1935 | 2744 | +41.8% | 1260 | 1545 | +22.6% |
| Reasoning / Thinking Tokens | 868 | 1421 | +63.7% | 392 | 634 | +61.7% |
| Total Input Tokens | 383227 | 500238 | +30.5% | 293576 | 304021 | +3.6% |
| Cached Input Tokens | 353024 | 463360 | +31.3% | 268800 | 275456 | +2.5% |
| Uncached Input Tokens | 30203 | 36878 | +22.1% | 24776 | 28565 | +15.3% |
| Cost | 0.1988 | 0.2601 | +30.8% | 0.1498 | 0.1675 | +11.8% |
| Cached vs Uncached Token Ratio | 11.69:1 | 12.56:1 | +7.5% | 10.85:1 | 9.64:1 | -11.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ? ✗ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification may format and write Go files, potentially modifying tests despite the user’s explicit prohibition; its broad write scope is not authorized. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 36.05s | 164.83s | +357.3% | 60.26s | 130.12s | +115.9% |
| Process Start → First Event | 0.12s | 0.18s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 7.74s | 11.35s | — | 8.72s | 12.17s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 22 | +340.0% | 7 | 16 | +128.6% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 22 | +340.0% | 7 | 16 | +128.6% |
| Output Tokens | 1204 | 3504 | +191.0% | 1385 | 3242 | +134.1% |
| Reasoning / Thinking Tokens | 373 | 1534 | +311.3% | 526 | 1693 | +221.9% |
| Total Input Tokens | 185859 | 554799 | +198.5% | 288440 | 545656 | +89.2% |
| Cached Input Tokens | 164096 | 513280 | +212.8% | 263680 | 503296 | +90.9% |
| Uncached Input Tokens | 21763 | 41519 | +90.8% | 24760 | 42360 | +71.1% |
| Cost | 0.1151 | 0.2951 | +156.3% | 0.1517 | 0.2934 | +93.4% |
| Cached vs Uncached Token Ratio | 7.54:1 | 12.36:1 | +64.0% | 10.65:1 | 11.88:1 | +11.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Dispatcher.Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517187432801000/audit/sink.go: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_lookup (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): read Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519416721828000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519416721828000/audit: is a directory |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): read Dispatcher in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519416721828000/audit: read /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519416721828000/audit: is a directory |
| 7 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_batch (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.52s | 108.66s | +124.0% | 46.79s | 85.08s | +81.8% |
| Process Start → First Event | 0.13s | 0.15s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 9.16s | 10.38s | — | 6.61s | 14.35s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 17 | +240.0% | 7 | 12 | +71.4% |
| Initial Load / Discovery Turns | 1 | 3 | +200.0% | 2 | 0 | -100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 17 | +240.0% | 7 | 12 | +71.4% |
| Output Tokens | 1235 | 1958 | +58.5% | 1746 | 1793 | +2.7% |
| Reasoning / Thinking Tokens | 273 | 633 | +131.9% | 427 | 813 | +90.4% |
| Total Input Tokens | 138260 | 518732 | +275.2% | 153130 | 409096 | +167.2% |
| Cached Input Tokens | 122368 | 473600 | +287.0% | 128256 | 376576 | +193.6% |
| Uncached Input Tokens | 15892 | 45132 | +184.0% | 24874 | 32520 | +30.7% |
| Cost | 0.0892 | 0.2636 | +195.6% | 0.1214 | 0.2080 | +71.3% |
| Cached vs Uncached Token Ratio | 7.70:1 | 10.49:1 | +36.3% | 5.16:1 | 11.58:1 | +124.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_lookup (✓ ✗): resolve health in audit/summary.go: symbol not found |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517756961474000/audit/normalize.go: symbol not found |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_batch (✓ ✓) |
| 12 | — | semedit/semantic_organize_imports (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 41.15s | 64.71s | +57.3% | 44.47s | 89.54s | +101.4% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 9.00s | 8.89s | — | 6.27s | 10.04s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 13 | +160.0% | 9 | 13 | +44.4% |
| Initial Load / Discovery Turns | 1 | 4 | +300.0% | 4 | 3 | -25.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 13 | +160.0% | 9 | 13 | +44.4% |
| Output Tokens | 1464 | 1413 | -3.5% | 1423 | 1641 | +15.3% |
| Reasoning / Thinking Tokens | 547 | 519 | -5.1% | 440 | 587 | +33.4% |
| Total Input Tokens | 224020 | 284137 | +26.8% | 215301 | 429183 | +99.3% |
| Cached Input Tokens | 200704 | 251648 | +25.4% | 191488 | 397824 | +107.8% |
| Uncached Input Tokens | 23316 | 32489 | +39.3% | 23813 | 31359 | +31.7% |
| Cost | 0.1336 | 0.1683 | +26.0% | 0.1307 | 0.2057 | +57.4% |
| Cached vs Uncached Token Ratio | 8.61:1 | 7.75:1 | -10.0% | 8.04:1 | 12.69:1 | +57.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | — | semedit/semantic_outline (✓ ✓) |
| 7 | — | semedit/semantic_outline (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_batch (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ? ✗ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 49.99s | 75.93s | +51.9% | 44.76s | 168.26s | +275.9% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 13.82s | 7.99s | — | 5.50s | 12.02s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 4 | 13 | +225.0% | 7 | 23 | +228.6% |
| Initial Load / Discovery Turns | 0 | 0 | — | 1 | 3 | +200.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 4 | 13 | +225.0% | 7 | 23 | +228.6% |
| Output Tokens | 1558 | 1931 | +23.9% | 1404 | 3788 | +169.8% |
| Reasoning / Thinking Tokens | 718 | 933 | +29.9% | 503 | 1446 | +187.5% |
| Total Input Tokens | 228965 | 283286 | +23.7% | 191467 | 733767 | +283.2% |
| Cached Input Tokens | 204800 | 249856 | +22.0% | 176896 | 684288 | +286.8% |
| Uncached Input Tokens | 24165 | 33430 | +38.3% | 14571 | 49479 | +239.6% |
| Cost | 0.1401 | 0.1818 | +29.8% | 0.1045 | 0.3602 | +244.7% |
| Cached vs Uncached Token Ratio | 8.48:1 | 7.47:1 | -11.8% | 12.14:1 | 13.83:1 | +13.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790516982134538000/audit/normalize.go: symbol not found |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_batch (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/report_feedback (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_replace_body", "status": "error", "error": "replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790519184129961000/service/dispatch.go: symbol not found" } ], "final_diff": "" } |
| 12 | — | semedit/report_feedback (✓ ✓) |
| 13 | — | semedit/semantic_batch (✓ ✓) |
| 14 | — | ? ✗ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 33.23s | 65.42s | +96.8% | 43.43s | 100.63s | +131.7% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 5.56s | 8.05s | — | 5.13s | 7.14s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 12 | +140.0% | 7 | 15 | +114.3% |
| Initial Load / Discovery Turns | 1 | 0 | -100.0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 12 | +140.0% | 7 | 15 | +114.3% |
| Output Tokens | 1162 | 1543 | +32.8% | 1423 | 1609 | +13.1% |
| Reasoning / Thinking Tokens | 350 | 557 | +59.1% | 470 | 625 | +33.0% |
| Total Input Tokens | 156623 | 270710 | +72.8% | 261139 | 345532 | +32.3% |
| Cached Input Tokens | 136704 | 235520 | +72.3% | 236288 | 310784 | +31.5% |
| Uncached Input Tokens | 19919 | 35190 | +76.7% | 24851 | 34748 | +39.8% |
| Cost | 0.1029 | 0.1731 | +68.3% | 0.1449 | 0.1925 | +32.9% |
| Cached vs Uncached Token Ratio | 6.86:1 | 6.69:1 | -2.5% | 9.51:1 | 8.94:1 | -5.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_batch (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 58.72s | 178.41s | +203.8% | 45.64s | 116.12s | +154.4% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 11.97s | 13.85s | — | 10.91s | 10.10s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 22 | +266.7% | 5 | 12 | +140.0% |
| Initial Load / Discovery Turns | 1 | 3 | +200.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 22 | +266.7% | 5 | 12 | +140.0% |
| Output Tokens | 2094 | 2678 | +27.9% | 1440 | 2486 | +72.6% |
| Reasoning / Thinking Tokens | 672 | 1376 | +104.8% | 683 | 1352 | +98.0% |
| Total Input Tokens | 286987 | 638416 | +122.5% | 255375 | 381232 | +49.3% |
| Cached Input Tokens | 262656 | 590080 | +124.7% | 231168 | 348672 | +50.8% |
| Uncached Input Tokens | 24331 | 48336 | +98.7% | 24207 | 32560 | +34.5% |
| Cost | 0.1611 | 0.3190 | +98.1% | 0.1448 | 0.2165 | +49.5% |
| Cached vs Uncached Token Ratio | 10.80:1 | 12.21:1 | +13.1% | 9.55:1 | 10.71:1 | +12.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_outline (✓ ✓) |
| 8 | — | semedit/semantic_outline (✓ ✓) |
| 9 | — | semedit/semantic_outline (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_insert_function (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517576886265000/service/dispatch.go: symbol not found |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
| 21 | — | ✓ ✓ |
| 22 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✗): param "file" is required: invalid operation parameters |
| 6 | — | semedit/semantic_replace_body (✓ ✗): param "file" is required: invalid operation parameters |
| 7 | — | semedit/semantic_batch (✓ ✓) |
| 8 | — | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 43.43s | 243.47s | +460.6% | 55.04s | 61.11s | +11.0% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 5.45s | 21.57s | — | 11.62s | 5.93s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 20 | +185.7% | 7 | 12 | +71.4% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 20 | +185.7% | 7 | 12 | +71.4% |
| Output Tokens | 1399 | 3153 | +125.4% | 1586 | 1576 | -0.6% |
| Reasoning / Thinking Tokens | 445 | 1841 | +313.7% | 352 | 506 | +43.8% |
| Total Input Tokens | 197286 | 458607 | +132.5% | 219416 | 285322 | +30.0% |
| Cached Input Tokens | 178176 | 416768 | +133.9% | 201216 | 249856 | +24.2% |
| Uncached Input Tokens | 19110 | 41839 | +118.9% | 18200 | 35466 | +94.9% |
| Cost | 0.1154 | 0.2712 | +135.1% | 0.1200 | 0.1772 | +47.6% |
| Cached vs Uncached Token Ratio | 9.32:1 | 9.96:1 | +6.8% | 11.06:1 | 7.04:1 | -36.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✗): unknown param "scope"; supported keys are [bash-bin, bash_bin, ghc-bin, ghc-version, ghc_bin, ghc_version, haskell-standalone, hls-bin, hls-version, hls_bin, hls_version, import-maven, import_maven, include-tests, include-unexported, include_tests, include_unexported, java-bin, java-version, java_bin, java_version, jdtls-home, jdtls_home, kind, kinds, kotlin-bin, kotlin_bin, language, make-bin, make_bin, metals-bin, metals-home, metals_bin, metals_home, path, standalone_haskell, trust-workspace, trust_works |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 8 | — | ✓ ✓ |
| 9 | — | ? ✗ |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | semedit/semantic_batch (✓ ✓) |
| 13 | — | semedit/semantic_organize_imports (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_batch (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 43.75s | 102.50s | +134.3% | 48.52s | 95.76s | +97.4% |
| Process Start → First Event | 0.17s | 0.13s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 9.90s | 10.49s | — | 5.60s | 7.76s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 18 | +260.0% | 7 | 15 | +114.3% |
| Initial Load / Discovery Turns | 0 | 3 | — | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 18 | +260.0% | 7 | 15 | +114.3% |
| Output Tokens | 1366 | 1941 | +42.1% | 1696 | 2198 | +29.6% |
| Reasoning / Thinking Tokens | 460 | 716 | +55.7% | 457 | 1063 | +132.6% |
| Total Input Tokens | 259839 | 415835 | +60.0% | 174580 | 424031 | +142.9% |
| Cached Input Tokens | 235264 | 364544 | +55.0% | 157696 | 389888 | +147.2% |
| Uncached Input Tokens | 24575 | 51291 | +108.7% | 16884 | 34143 | +102.2% |
| Cost | 0.1431 | 0.2526 | +76.5% | 0.1085 | 0.2236 | +106.0% |
| Cached vs Uncached Token Ratio | 9.57:1 | 7.11:1 | -25.8% | 9.34:1 | 11.42:1 | +22.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
prefer_discover_semedit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 56.45s | 77.36s | +37.0% | 52.13s | 89.25s | +71.2% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 13.80s | 11.00s | — | 5.14s | 8.48s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 12 | +100.0% | 10 | 14 | +40.0% |
| Initial Load / Discovery Turns | 0 | 0 | — | 2 | 0 | -100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 12 | +100.0% | 10 | 14 | +40.0% |
| Output Tokens | 1933 | 1760 | -8.9% | 1997 | 1100 | -44.9% |
| Reasoning / Thinking Tokens | 586 | 574 | -2.0% | 506 | 0 | -100.0% |
| Total Input Tokens | 291760 | 329546 | +13.0% | 174286 | 517618 | +197.0% |
| Cached Input Tokens | 265728 | 301312 | +13.4% | 157696 | 479488 | +204.1% |
| Uncached Input Tokens | 26032 | 28234 | +8.5% | 16590 | 38130 | +129.8% |
| Cost | 0.1630 | 0.1751 | +7.4% | 0.1122 | 0.2289 | +104.1% |
| Cached vs Uncached Token Ratio | 10.21:1 | 10.67:1 | +4.5% | 9.51:1 | 12.58:1 | +32.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 36.92s | 95.22s | +157.9% | 101.08s | 86.57s | -14.4% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.51s | — |
| First Event → First Tool Call | 9.00s | 10.54s | — | 12.15s | 11.26s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 3 | 10 | +233.3% | 7 | 15 | +114.3% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 2 | 4 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 3 | 10 | +233.3% | 7 | 15 | +114.3% |
| Output Tokens | 1280 | 1985 | +55.1% | 1914 | 2218 | +15.9% |
| Reasoning / Thinking Tokens | 549 | 712 | +29.7% | 641 | 875 | +36.5% |
| Total Input Tokens | 129629 | 444594 | +243.0% | 204206 | 469316 | +129.8% |
| Cached Input Tokens | 115200 | 413952 | +259.3% | 186112 | 432896 | +132.6% |
| Uncached Input Tokens | 14429 | 30642 | +112.4% | 18094 | 36420 | +101.3% |
| Cost | 0.0877 | 0.2138 | +143.7% | 0.1237 | 0.2379 | +92.3% |
| Cached vs Uncached Token Ratio | 7.98:1 | 13.51:1 | +69.2% | 10.29:1 | 11.89:1 | +15.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | — | semedit/semantic_scaffold_file (✓ ✗): file already exists: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790516835578685000/audit/normalize.go |
| 5 | — | semedit/report_feedback (✗ ?): This action was rejected due to unacceptable risk. Reason: The feedback call would transmit a private local path and project/tool details to an unspecified external feedback destination, which the user did not authorize. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains bloc |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_batch (✓ ✓) |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ? ✗ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 55.86s | 107.43s | +92.3% | 55.64s | 68.75s | +23.6% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 9.02s | 7.48s | — | 5.44s | 5.87s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 16 | +166.7% | 7 | 12 | +71.4% |
| Initial Load / Discovery Turns | 0 | 3 | — | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 16 | +166.7% | 7 | 12 | +71.4% |
| Output Tokens | 1474 | 2142 | +45.3% | 1882 | 1465 | -22.2% |
| Reasoning / Thinking Tokens | 560 | 809 | +44.5% | 605 | 440 | -27.3% |
| Total Input Tokens | 188918 | 446697 | +136.5% | 215212 | 273081 | +26.9% |
| Cached Input Tokens | 172800 | 408576 | +136.4% | 191488 | 243456 | +27.1% |
| Uncached Input Tokens | 16118 | 38121 | +136.5% | 23724 | 29625 | +24.9% |
| Cost | 0.1089 | 0.2343 | +115.1% | 0.1383 | 0.1587 | +14.8% |
| Cached vs Uncached Token Ratio | 10.72:1 | 10.72:1 | -0.0% | 8.07:1 | 8.22:1 | +1.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517531788241000/audit/normalize.go: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_batch (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 8 | — | ? ✗ |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 54.67s | 77.78s | +42.3% | 60.96s | 120.64s | +97.9% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 9.93s | 13.02s | — | 13.24s | 12.27s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 11 | +57.1% | 5 | 17 | +240.0% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 11 | +57.1% | 5 | 17 | +240.0% |
| Output Tokens | 1465 | 1599 | +9.1% | 2262 | 2723 | +20.4% |
| Reasoning / Thinking Tokens | 488 | 567 | +16.2% | 927 | 1081 | +16.6% |
| Total Input Tokens | 289835 | 366847 | +26.6% | 295365 | 574456 | +94.5% |
| Cached Input Tokens | 263680 | 329728 | +25.0% | 267776 | 532736 | +98.9% |
| Uncached Input Tokens | 26155 | 37119 | +41.9% | 27589 | 41720 | +51.2% |
| Cost | 0.1557 | 0.2023 | +29.9% | 0.1758 | 0.2850 | +62.2% |
| Cached vs Uncached Token Ratio | 10.08:1 | 8.88:1 | -11.9% | 9.71:1 | 12.77:1 | +31.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ? ✗ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 66.53s | 72.95s | +9.6% | 54.61s | 109.42s | +100.4% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.14s | — |
| First Event → First Tool Call | 11.65s | 9.35s | — | 8.14s | 19.80s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 10 | +42.9% | 7 | 11 | +57.1% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 10 | +42.9% | 7 | 11 | +57.1% |
| Output Tokens | 2183 | 1591 | -27.1% | 1877 | 1962 | +4.5% |
| Reasoning / Thinking Tokens | 606 | 665 | +9.7% | 448 | 927 | +106.9% |
| Total Input Tokens | 217698 | 286884 | +31.8% | 183350 | 459493 | +150.6% |
| Cached Input Tokens | 198400 | 262656 | +32.4% | 167680 | 421632 | +151.5% |
| Uncached Input Tokens | 19298 | 24228 | +25.5% | 15670 | 37861 | +141.6% |
| Cost | 0.1327 | 0.1544 | +16.4% | 0.1102 | 0.2362 | +114.4% |
| Cached vs Uncached Token Ratio | 10.28:1 | 10.84:1 | +5.4% | 10.70:1 | 11.14:1 | +4.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | ? ✗ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 47.56s | 89.53s | +88.2% | 49.11s | 93.48s | +90.3% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 9.29s | 13.25s | — | 7.10s | 12.39s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 13 | +116.7% | 6 | 16 | +166.7% |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 13 | +116.7% | 6 | 16 | +166.7% |
| Output Tokens | 1543 | 1820 | +18.0% | 1398 | 2022 | +44.6% |
| Reasoning / Thinking Tokens | 661 | 722 | +9.2% | 394 | 769 | +95.2% |
| Total Input Tokens | 264036 | 311850 | +18.1% | 186432 | 658352 | +253.1% |
| Cached Input Tokens | 239360 | 276224 | +15.4% | 168960 | 614144 | +263.5% |
| Uncached Input Tokens | 24676 | 35626 | +44.4% | 17472 | 44208 | +153.0% |
| Cost | 0.1491 | 0.1899 | +27.4% | 0.1083 | 0.2989 | +176.0% |
| Cached vs Uncached Token Ratio | 9.70:1 | 7.75:1 | -20.1% | 9.67:1 | 13.89:1 | +43.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ? ✗ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 70.85s | 73.97s | +4.4% | 68.89s | 72.16s | +4.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 8.72s | 7.75s | — | 12.31s | 9.98s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 13 | +85.7% | 7 | 9 | +28.6% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 13 | +85.7% | 7 | 9 | +28.6% |
| Output Tokens | 1700 | 1854 | +9.1% | 1766 | 1711 | -3.1% |
| Reasoning / Thinking Tokens | 753 | 695 | -7.7% | 829 | 656 | -20.9% |
| Total Input Tokens | 242568 | 318239 | +31.2% | 321555 | 378951 | +17.8% |
| Cached Input Tokens | 219904 | 289280 | +31.5% | 294144 | 352000 | +19.7% |
| Uncached Input Tokens | 22664 | 28959 | +27.8% | 27411 | 26951 | -1.7% |
| Cost | 0.1423 | 0.1766 | +24.1% | 0.1745 | 0.1850 | +6.0% |
| Cached vs Uncached Token Ratio | 9.70:1 | 9.99:1 | +3.0% | 10.73:1 | 13.06:1 | +21.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | ? ✗ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 55.11s | 73.86s | +34.0% | 54.62s | 84.19s | +54.1% |
| Process Start → First Event | 0.47s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 9.58s | 10.08s | — | 9.97s | 6.67s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 11 | +83.3% | 7 | 14 | +100.0% |
| Initial Load / Discovery Turns | 1 | 0 | -100.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 11 | +83.3% | 7 | 14 | +100.0% |
| Output Tokens | 2099 | 1874 | -10.7% | 1779 | 1831 | +2.9% |
| Reasoning / Thinking Tokens | 544 | 897 | +64.9% | 665 | 671 | +0.9% |
| Total Input Tokens | 193188 | 351098 | +81.7% | 290513 | 341421 | +17.5% |
| Cached Input Tokens | 178944 | 322560 | +80.3% | 265728 | 304896 | +14.7% |
| Uncached Input Tokens | 14244 | 28538 | +100.4% | 24785 | 36525 | +47.4% |
| Cost | 0.1134 | 0.1866 | +64.6% | 0.1589 | 0.1988 | +25.1% |
| Cached vs Uncached Token Ratio | 12.56:1 | 11.30:1 | -10.0% | 10.72:1 | 8.35:1 | -22.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_verify (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ? ✗ |
| 14 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 68.96s | 124.06s | +79.9% | 88.62s | 86.16s | -2.8% |
| Process Start → First Event | 0.15s | 0.14s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 13.83s | 14.15s | — | 15.98s | 10.28s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 17 | +112.5% | 8 | 9 | +12.5% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 0 | 1 | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 17 | +112.5% | 8 | 9 | +12.5% |
| Output Tokens | 2267 | 2423 | +6.9% | 2460 | 1708 | -30.6% |
| Reasoning / Thinking Tokens | 689 | 977 | +41.8% | 503 | 733 | +45.7% |
| Total Input Tokens | 296540 | 470143 | +58.5% | 245594 | 240579 | -2.0% |
| Cached Input Tokens | 273408 | 428800 | +56.8% | 226560 | 206592 | -8.8% |
| Uncached Input Tokens | 23132 | 41343 | +78.7% | 19034 | 33987 | +78.6% |
| Cost | 0.1631 | 0.2531 | +55.1% | 0.1413 | 0.1671 | +18.3% |
| Cached vs Uncached Token Ratio | 11.82:1 | 10.37:1 | -12.2% | 11.90:1 | 6.08:1 | -48.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ? ✗ | semedit/semantic_batch (✓ ✓) |
| 6 | ✓ ✓ | ? ✗ |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 51.19s | 82.09s | +60.4% | 50.96s | 61.85s | +21.4% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.14s | 0.15s | — |
| First Event → First Tool Call | 6.80s | 11.88s | — | 13.19s | 8.06s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 10 | +42.9% | 6 | 10 | +66.7% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 0 | 1 | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 10 | +42.9% | 6 | 10 | +66.7% |
| Output Tokens | 1510 | 1957 | +29.6% | 1314 | 1662 | +26.5% |
| Reasoning / Thinking Tokens | 386 | 651 | +68.7% | 500 | 789 | +57.8% |
| Total Input Tokens | 164027 | 324493 | +97.8% | 290345 | 262429 | -9.6% |
| Cached Input Tokens | 149504 | 297216 | +98.8% | 264704 | 236288 | -10.7% |
| Uncached Input Tokens | 14523 | 27277 | +87.8% | 25641 | 26141 | +2.0% |
| Cost | 0.0974 | 0.1751 | +79.8% | 0.1530 | 0.1551 | +1.4% |
| Cached vs Uncached Token Ratio | 10.29:1 | 10.90:1 | +5.8% | 10.32:1 | 9.04:1 | -12.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | ? ✗ |
| 8 | — | semedit/semantic_verify (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ? ✗ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 55.49s | 93.09s | +67.8% | 37.98s | 102.04s | +168.7% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 11.94s | 10.46s | — | 5.69s | 8.83s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 14 | +133.3% | 6 | 11 | +83.3% |
| Initial Load / Discovery Turns | 0 | 1 | — | 1 | 1 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 14 | +133.3% | 6 | 11 | +83.3% |
| Output Tokens | 1908 | 2349 | +23.1% | 1236 | 2274 | +84.0% |
| Reasoning / Thinking Tokens | 530 | 1344 | +153.6% | 355 | 982 | +176.6% |
| Total Input Tokens | 294700 | 338383 | +14.8% | 141506 | 408710 | +188.8% |
| Cached Input Tokens | 267776 | 305664 | +14.1% | 127232 | 380416 | +199.0% |
| Uncached Input Tokens | 26924 | 32719 | +21.5% | 14274 | 28294 | +98.2% |
| Cost | 0.1647 | 0.2044 | +24.1% | 0.0874 | 0.2065 | +136.4% |
| Cached vs Uncached Token Ratio | 9.95:1 | 9.34:1 | -6.1% | 8.91:1 | 13.45:1 | +50.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790517058153206000/audit/normalize.go: symbol not found |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_batch (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_verify (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 7 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✗): unknown param "body"; supported keys are [access, access_modifier, auto-organize-imports, auto_organize_imports, file, placement, source, target, target_symbol]: invalid operation parameters |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/report_feedback (✗ ?): This action was rejected due to unacceptable risk. Reason: The feedback call would transmit implementation details and tool/error context to an unspecified feedback destination, which the user did not authorize and is unnecessary for completing the requested local changes. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without askin |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |