Benchmark run gpt-6-luna-all-r10-c5-prescriptive-20260927-retry
This page contains exactly the publishable benchmark observations recorded in run gpt-6-luna-all-r10-c5-prescriptive-20260927-retry; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: task-04-insert-public
Target: codex/gpt-6-luna/medium
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 14.59s | 35.94s | +146.4% | 23.48s | 23.68s | +0.9% |
| Process Start → First Event | 0.57s | 0.57s | — | 0.14s | 0.14s | — |
| First Event → First Tool Call | 4.94s | 5.64s | — | 4.11s | 8.31s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 2 | 1 | -50.0% |
| Internal Tool Cycles | 2 | 7 | +250.0% | 5 | 4 | -20.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 7 | +250.0% | 5 | 4 | -20.0% |
| Output Tokens | 307 | 778 | +153.4% | 804 | 359 | -55.3% |
| Reasoning / Thinking Tokens | 73 | 289 | +295.9% | 185 | 62 | -66.5% |
| Total Input Tokens | 76207 | 204078 | +167.8% | 192269 | 137896 | -28.3% |
| Cached Input Tokens | 68608 | 181504 | +164.6% | 171520 | 111104 | -35.2% |
| Uncached Input Tokens | 7599 | 22574 | +197.1% | 20749 | 26792 | +29.1% |
| Cost | 0.0409 | 0.1151 | +181.5% | 0.1071 | 0.1000 | -6.6% |
| Cached vs Uncached Token Ratio | 9.03:1 | 8.04:1 | -10.9% | 8.27:1 | 4.15:1 | -49.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_outline (✓ ✓) |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | — |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 22.11s | 27.15s | +22.8% | 16.01s | 29.31s | +83.1% |
| Process Start → First Event | 0.12s | 0.14s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 4.65s | 4.86s | — | 5.64s | 4.85s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 5 | 0% | 2 | 5 | +150.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 5 | 0% | 2 | 5 | +150.0% |
| Output Tokens | 703 | 586 | -16.6% | 433 | 432 | -0.2% |
| Reasoning / Thinking Tokens | 116 | 223 | +92.2% | 85 | 56 | -34.1% |
| Total Input Tokens | 114983 | 155840 | +35.5% | 76025 | 154681 | +103.5% |
| Cached Input Tokens | 97792 | 134656 | +37.7% | 66560 | 134656 | +102.3% |
| Uncached Input Tokens | 17191 | 21184 | +23.2% | 9465 | 20025 | +111.6% |
| Cost | 0.0777 | 0.0967 | +24.6% | 0.0468 | 0.0898 | +92.0% |
| Cached vs Uncached Token Ratio | 5.69:1 | 6.36:1 | +11.7% | 7.03:1 | 6.72:1 | -4.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | — | semedit/semantic_outline (✓ ✓) |
| 4 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 17.32s | 21.70s | +25.3% | 20.14s | 32.11s | +59.5% |
| Process Start → First Event | 0.59s | 0.57s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 5.58s | 6.97s | — | 5.12s | 4.22s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 2 | 3 | +50.0% | 4 | 5 | +25.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 3 | +50.0% | 4 | 5 | +25.0% |
| Output Tokens | 364 | 353 | -3.0% | 730 | 677 | -7.3% |
| Reasoning / Thinking Tokens | 25 | 43 | +72.0% | 146 | 186 | +27.4% |
| Total Input Tokens | 76195 | 99778 | +31.0% | 97753 | 124785 | +27.7% |
| Cached Input Tokens | 68608 | 74496 | +8.6% | 86784 | 110080 | +26.8% |
| Uncached Input Tokens | 7587 | 25282 | +233.2% | 10969 | 14705 | +34.1% |
| Cost | 0.0410 | 0.0868 | +111.7% | 0.0601 | 0.0751 | +25.0% |
| Cached vs Uncached Token Ratio | 9.04:1 | 2.95:1 | -67.4% | 7.91:1 | 7.49:1 | -5.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 18.18s | 27.39s | +50.7% | 13.86s | 29.74s | +114.6% |
| Process Start → First Event | 0.57s | 0.16s | — | 0.12s | 0.14s | — |
| First Event → First Tool Call | 4.69s | 6.75s | — | 4.29s | 4.09s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 3 | 5 | +66.7% | 2 | 5 | +150.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 3 | 5 | +66.7% | 2 | 5 | +150.0% |
| Output Tokens | 483 | 535 | +10.8% | 389 | 519 | +33.4% |
| Reasoning / Thinking Tokens | 151 | 126 | -16.6% | 100 | 106 | +6.0% |
| Total Input Tokens | 76171 | 159669 | +109.6% | 76000 | 209923 | +176.2% |
| Cached Input Tokens | 68608 | 129536 | +88.8% | 66560 | 167936 | +152.3% |
| Uncached Input Tokens | 7563 | 30133 | +298.4% | 9440 | 41987 | +344.8% |
| Cost | 0.0440 | 0.1160 | +163.7% | 0.0464 | 0.1548 | +233.9% |
| Cached vs Uncached Token Ratio | 9.07:1 | 4.30:1 | -52.6% | 7.05:1 | 4.00:1 | -43.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | — | semedit/semantic_outline (✓ ✓) |
| 4 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 13.00s | 28.89s | +122.3% | 13.11s | 32.69s | +149.2% |
| Process Start → First Event | 0.17s | 0.15s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 4.52s | 9.98s | — | 3.91s | 5.00s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 2 | 3 | +50.0% | 2 | 6 | +200.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 3 | +50.0% | 2 | 6 | +200.0% |
| Output Tokens | 283 | 638 | +125.4% | 262 | 629 | +140.1% |
| Reasoning / Thinking Tokens | 0 | 235 | — | 75 | 144 | +92.0% |
| Total Input Tokens | 75659 | 166311 | +119.8% | 75668 | 205628 | +171.8% |
| Cached Input Tokens | 66560 | 142848 | +114.6% | 66560 | 180480 | +171.2% |
| Uncached Input Tokens | 9099 | 23463 | +157.9% | 9108 | 25148 | +176.1% |
| Cost | 0.0429 | 0.1053 | +145.3% | 0.0436 | 0.1177 | +169.7% |
| Cached vs Uncached Token Ratio | 7.32:1 | 6.09:1 | -16.8% | 7.31:1 | 7.18:1 | -1.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_outline (✓ ✓) |
| 4 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 11.52s | 22.98s | +99.5% | 31.87s | 28.39s | -10.9% |
| Process Start → First Event | 0.13s | 0.15s | — | 0.14s | 0.13s | — |
| First Event → First Tool Call | 4.03s | 5.41s | — | 4.68s | 6.12s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 2 | 1 | -50.0% |
| Internal Tool Cycles | 2 | 4 | +100.0% | 6 | 6 | 0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 4 | +100.0% | 6 | 6 | 0% |
| Output Tokens | 256 | 446 | +74.2% | 1547 | 429 | -72.3% |
| Reasoning / Thinking Tokens | 51 | 139 | +172.5% | 218 | 74 | -66.1% |
| Total Input Tokens | 75703 | 96322 | +27.2% | 231262 | 158537 | -31.4% |
| Cached Input Tokens | 66560 | 79872 | +20.0% | 209920 | 141568 | -32.6% |
| Uncached Input Tokens | 9143 | 16450 | +79.9% | 21342 | 16969 | -20.5% |
| Cost | 0.0433 | 0.0684 | +57.9% | 0.1279 | 0.0841 | -34.2% |
| Cached vs Uncached Token Ratio | 7.28:1 | 4.86:1 | -33.3% | 9.84:1 | 8.34:1 | -15.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | — | semedit/semantic_outline (✓ ✓) |
| 4 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 7 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 11.95s | 20.10s | +68.3% | 21.23s | 36.73s | +73.0% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 3.94s | 5.08s | — | 5.39s | 4.54s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 2 | 3 | +50.0% | 4 | 6 | +50.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 3 | +50.0% | 4 | 6 | +50.0% |
| Output Tokens | 279 | 330 | +18.3% | 671 | 620 | -7.6% |
| Reasoning / Thinking Tokens | 0 | 50 | — | 98 | 199 | +103.1% |
| Total Input Tokens | 76664 | 105190 | +37.2% | 95753 | 204944 | +114.0% |
| Cached Input Tokens | 49408 | 89856 | +81.9% | 85760 | 183296 | +113.7% |
| Uncached Input Tokens | 27256 | 15334 | -43.7% | 9993 | 21648 | +116.6% |
| Cost | 0.0840 | 0.0655 | -21.9% | 0.0560 | 0.1102 | +96.6% |
| Cached vs Uncached Token Ratio | 1.81:1 | 5.86:1 | +223.3% | 8.58:1 | 8.47:1 | -1.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | semedit/semantic_outline (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 16.79s | 32.87s | +95.8% | 19.51s | 19.21s | -1.5% |
| Process Start → First Event | 0.52s | 0.53s | — | 0.12s | 0.17s | — |
| First Event → First Tool Call | 4.80s | 4.17s | — | 4.19s | 4.90s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 3 | 7 | +133.3% | 4 | 3 | -25.0% |
| Initial Load / Discovery Turns | 2 | 3 | +50.0% | 4 | 1 | -75.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 3 | 7 | +133.3% | 4 | 3 | -25.0% |
| Output Tokens | 525 | 736 | +40.2% | 431 | 350 | -18.8% |
| Reasoning / Thinking Tokens | 95 | 299 | +214.7% | 73 | 50 | -31.5% |
| Total Input Tokens | 76313 | 158463 | +107.6% | 115215 | 101144 | -12.2% |
| Cached Input Tokens | 68608 | 140544 | +104.9% | 104960 | 87808 | -16.3% |
| Uncached Input Tokens | 7705 | 17919 | +132.6% | 10255 | 13336 | +30.0% |
| Cost | 0.0442 | 0.0929 | +110.3% | 0.0582 | 0.0603 | +3.6% |
| Cached vs Uncached Token Ratio | 8.90:1 | 7.84:1 | -11.9% | 10.24:1 | 6.58:1 | -35.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | — | semedit/semantic_outline (✓ ✓) |
| 5 | — | semedit/semantic_insert_function (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 4 | ✓ ✓ | — |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 23.70s | 41.18s | +73.8% | 28.31s | 36.18s | +27.8% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 4.12s | 5.48s | — | 3.78s | 6.01s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 2 | 1 | -50.0% | 1 | 1 | 0% |
| Internal Tool Cycles | 4 | 6 | +50.0% | 7 | 6 | -14.3% |
| Initial Load / Discovery Turns | 4 | 1 | -75.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 4 | 6 | +50.0% | 7 | 6 | -14.3% |
| Output Tokens | 642 | 673 | +4.8% | 865 | 548 | -36.6% |
| Reasoning / Thinking Tokens | 115 | 271 | +135.7% | 181 | 139 | -23.2% |
| Total Input Tokens | 193322 | 180159 | -6.8% | 155110 | 177444 | +14.4% |
| Cached Input Tokens | 173568 | 149504 | -13.9% | 142336 | 159744 | +12.2% |
| Uncached Input Tokens | 19754 | 30655 | +55.2% | 12774 | 17700 | +38.6% |
| Cost | 0.1022 | 0.1258 | +23.1% | 0.0806 | 0.0928 | +15.1% |
| Cached vs Uncached Token Ratio | 8.79:1 | 4.88:1 | -44.5% | 11.14:1 | 9.03:1 | -19.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | — | ✓ ✓ |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | — |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-04-insert-public |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_04_insert_public.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package api
type Server struct {
Port int
}
func (s *Server) Start() {}
func (s *Server) internalRun() {}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 11.91s | 22.18s | +86.2% | 13.71s | 35.94s | +162.1% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.12s | 0.16s | — |
| First Event → First Tool Call | 4.17s | 4.80s | — | 4.21s | 5.69s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 2 | 4 | +100.0% | 3 | 7 | +133.3% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 2 | 4 | +100.0% | 3 | 7 | +133.3% |
| Output Tokens | 263 | 419 | +59.3% | 306 | 654 | +113.7% |
| Reasoning / Thinking Tokens | 0 | 105 | — | 45 | 100 | +122.2% |
| Total Input Tokens | 75552 | 129499 | +71.4% | 94929 | 117868 | +24.2% |
| Cached Input Tokens | 66560 | 113152 | +70.0% | 84736 | 101120 | +19.3% |
| Uncached Input Tokens | 8992 | 16347 | +81.8% | 10193 | 16748 | +64.3% |
| Cost | 0.0424 | 0.0757 | +78.5% | 0.0511 | 0.0766 | +50.0% |
| Cached vs Uncached Token Ratio | 7.40:1 | 6.92:1 | -6.5% | 8.31:1 | 6.04:1 | -27.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_insert_function (✓ ✓) |
| 4 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_outline (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_insert_function (✓ ✓) |
Test case: task-07-generate-template-main
Target: codex/gpt-6-luna/medium
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 19.74s | 72.44s | +266.9% | — | — | — |
| Process Start → First Event | 0.14s | 0.14s | — | — | — | — |
| First Event → First Tool Call | 5.58s | 6.42s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 12 | +300.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 12 | +300.0% | — | — | — |
| Output Tokens | 593 | 850 | +43.3% | — | — | — |
| Reasoning / Thinking Tokens | 182 | 0 | -100.0% | — | — | — |
| Total Input Tokens | 76315 | 300194 | +293.4% | — | — | — |
| Cached Input Tokens | 66560 | 275712 | +314.2% | — | — | — |
| Uncached Input Tokens | 9755 | 24482 | +151.0% | — | — | — |
| Cost | 0.0507 | 0.1408 | +177.5% | — | — | — |
| Cached vs Uncached Token Ratio | 6.82:1 | 11.26:1 | +65.1% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_organize_imports (✓ ✓) |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_organize_imports (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 15.82s | 43.20s | +173.0% | — | — | — |
| Process Start → First Event | 0.11s | 0.15s | — | — | — | — |
| First Event → First Tool Call | 3.83s | 4.53s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 8 | +100.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 2 | 0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 8 | +100.0% | — | — | — |
| Output Tokens | 493 | 1058 | +114.6% | — | — | — |
| Reasoning / Thinking Tokens | 14 | 261 | +1764.3% | — | — | — |
| Total Input Tokens | 96264 | 201744 | +109.6% | — | — | — |
| Cached Input Tokens | 85760 | 183040 | +113.4% | — | — | — |
| Uncached Input Tokens | 10504 | 18704 | +78.1% | — | — | — |
| Cost | 0.0540 | 0.1090 | +101.7% | — | — | — |
| Cached vs Uncached Token Ratio | 8.16:1 | 9.79:1 | +19.9% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 19 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_replace_body (✓ ✗): syntax error: expected '}', found 'EOF' |
| 5 | — | ? ✗ |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 51.55s | 33.36s | -35.3% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 17.55s | 6.62s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 7 | +133.3% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 7 | +133.3% | — | — | — |
| Output Tokens | 648 | 547 | -15.6% | — | — | — |
| Reasoning / Thinking Tokens | 234 | 0 | -100.0% | — | — | — |
| Total Input Tokens | 76354 | 157946 | +106.9% | — | — | — |
| Cached Input Tokens | 72704 | 142592 | +96.1% | — | — | — |
| Uncached Input Tokens | 3650 | 15354 | +320.7% | — | — | — |
| Cost | 0.0383 | 0.0809 | +111.0% | — | — | — |
| Cached vs Uncached Token Ratio | 19.92:1 | 9.29:1 | -53.4% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 30 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_organize_imports (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 17.78s | 28.02s | +57.6% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.96s | 6.00s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 5 | +150.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 2 | 0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 5 | +150.0% | — | — | — |
| Output Tokens | 537 | 685 | +27.6% | — | — | — |
| Reasoning / Thinking Tokens | 141 | 8 | -94.3% | — | — | — |
| Total Input Tokens | 76080 | 134391 | +76.6% | — | — | — |
| Cached Input Tokens | 66560 | 119296 | +79.2% | — | — | — |
| Uncached Input Tokens | 9520 | 15095 | +58.6% | — | — | — |
| Cost | 0.0489 | 0.0762 | +55.8% | — | — | — |
| Cached vs Uncached Token Ratio | 6.99:1 | 7.90:1 | +13.0% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 18.07s | 21.13s | +16.9% | — | — | — |
| Process Start → First Event | 0.13s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.08s | 5.47s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 4 | +33.3% | — | — | — |
| Initial Load / Discovery Turns | 2 | 2 | 0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 4 | +33.3% | — | — | — |
| Output Tokens | 566 | 416 | -26.5% | — | — | — |
| Reasoning / Thinking Tokens | 137 | 18 | -86.9% | — | — | — |
| Total Input Tokens | 76379 | 132875 | +74.0% | — | — | — |
| Cached Input Tokens | 66560 | 113408 | +70.4% | — | — | — |
| Uncached Input Tokens | 9819 | 19467 | +98.3% | — | — | — |
| Cost | 0.0500 | 0.0824 | +65.0% | — | — | — |
| Cached vs Uncached Token Ratio | 6.78:1 | 5.83:1 | -14.1% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 24.32s | 51.16s | +110.4% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 5.33s | 5.76s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 8 | +100.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 8 | +100.0% | — | — | — |
| Output Tokens | 761 | 856 | +12.5% | — | — | — |
| Reasoning / Thinking Tokens | 241 | 48 | -80.1% | — | — | — |
| Total Input Tokens | 96632 | 155652 | +61.1% | — | — | — |
| Cached Input Tokens | 85760 | 134656 | +57.0% | — | — | — |
| Uncached Input Tokens | 10872 | 20996 | +93.1% | — | — | — |
| Cost | 0.0611 | 0.0975 | +59.4% | — | — | — |
| Cached vs Uncached Token Ratio | 7.89:1 | 6.41:1 | -18.7% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | FAIL | PASS | — | — | — | — |
| Oracle L3: Clean Build | FAIL | PASS | — | — | — | — |
| Oracle L4: Verification Test | FAIL | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 31 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 24.24s | 26.55s | +9.5% | — | — | — |
| Process Start → First Event | 0.15s | 0.14s | — | — | — | — |
| First Event → First Tool Call | 7.67s | 5.56s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 4 | 0% | — | — | — |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 4 | 0% | — | — | — |
| Output Tokens | 633 | 742 | +17.2% | — | — | — |
| Reasoning / Thinking Tokens | 169 | 204 | +20.7% | — | — | — |
| Total Input Tokens | 97413 | 112889 | +15.9% | — | — | — |
| Cached Input Tokens | 86784 | 97024 | +11.8% | — | — | — |
| Uncached Input Tokens | 10629 | 15865 | +49.3% | — | — | — |
| Cost | 0.0583 | 0.0757 | +29.9% | — | — | — |
| Cached vs Uncached Token Ratio | 8.16:1 | 6.12:1 | -25.1% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | FAIL | — | — | — | — |
| Oracle L3: Clean Build | PASS | FAIL | — | — | — | — |
| Oracle L4: Verification Test | PASS | FAIL | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 27 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 21.10s | 48.89s | +131.7% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.58s | 6.59s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 7 | +133.3% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 7 | +133.3% | — | — | — |
| Output Tokens | 596 | 554 | -7.0% | — | — | — |
| Reasoning / Thinking Tokens | 215 | 0 | -100.0% | — | — | — |
| Total Input Tokens | 96183 | 155506 | +61.7% | — | — | — |
| Cached Input Tokens | 85760 | 139520 | +62.7% | — | — | — |
| Uncached Input Tokens | 10423 | 15986 | +53.4% | — | — | — |
| Cost | 0.0576 | 0.0818 | +41.9% | — | — | — |
| Cached vs Uncached Token Ratio | 8.23:1 | 8.73:1 | +6.1% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 19 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_organize_imports (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 18.77s | 35.31s | +88.1% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.58s | 4.66s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 6 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 2 | 0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 6 | +200.0% | — | — | — |
| Output Tokens | 599 | 668 | +11.5% | — | — | — |
| Reasoning / Thinking Tokens | 252 | 186 | -26.2% | — | — | — |
| Total Input Tokens | 76176 | 145612 | +91.2% | — | — | — |
| Cached Input Tokens | 66560 | 131328 | +97.3% | — | — | — |
| Uncached Input Tokens | 9616 | 14284 | +48.5% | — | — | — |
| Cost | 0.0513 | 0.0792 | +54.4% | — | — | — |
| Cached vs Uncached Token Ratio | 6.92:1 | 9.19:1 | +32.8% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | — | ? ✗ |
| 4 | — | semedit/semantic_lookup (✓ ✓) |
| 5 | — | semedit/semantic_replace_body (✓ ✓) |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
crypto_rand prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | crypto_rand |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 19.18s | 30.39s | +58.5% | — | — | — |
| Process Start → First Event | 0.18s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.59s | 4.64s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 5 | +25.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 5 | +25.0% | — | — | — |
| Output Tokens | 622 | 456 | -26.7% | — | — | — |
| Reasoning / Thinking Tokens | 32 | 0 | -100.0% | — | — | — |
| Total Input Tokens | 96067 | 134607 | +40.1% | — | — | — |
| Cached Input Tokens | 85760 | 120320 | +40.3% | — | — | — |
| Uncached Input Tokens | 10307 | 14287 | +38.6% | — | — | — |
| Cost | 0.0554 | 0.0715 | +29.1% | — | — | — |
| Cached vs Uncached Token Ratio | 8.32:1 | 8.42:1 | +1.2% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 11.14s | 30.90s | +177.4% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.04s | 5.29s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 5 | +150.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 5 | +150.0% | — | — | — |
| Output Tokens | 283 | 708 | +150.2% | — | — | — |
| Reasoning / Thinking Tokens | 0 | 301 | — | — | — | — |
| Total Input Tokens | 75783 | 138090 | +82.2% | — | — | — |
| Cached Input Tokens | 66560 | 122368 | +83.8% | — | — | — |
| Uncached Input Tokens | 9223 | 15722 | +70.5% | — | — | — |
| Cost | 0.0432 | 0.0825 | +90.8% | — | — | — |
| Cached vs Uncached Token Ratio | 7.22:1 | 7.78:1 | +7.8% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | — | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 18.73s | 57.02s | +204.4% | — | — | — |
| Process Start → First Event | 0.20s | 0.13s | — | — | — | — |
| First Event → First Tool Call | 4.29s | 7.29s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 8 | +100.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 8 | +100.0% | — | — | — |
| Output Tokens | 584 | 782 | +33.9% | — | — | — |
| Reasoning / Thinking Tokens | 166 | 143 | -13.9% | — | — | — |
| Total Input Tokens | 96135 | 204007 | +112.2% | — | — | — |
| Cached Input Tokens | 85760 | 172288 | +100.9% | — | — | — |
| Uncached Input Tokens | 10375 | 31719 | +205.7% | — | — | — |
| Cost | 0.0568 | 0.1339 | +136.0% | — | — | — |
| Cached vs Uncached Token Ratio | 8.27:1 | 5.43:1 | -34.3% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_replace_body (✓ ✓) |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 15.90s | 69.77s | +338.9% | — | — | — |
| Process Start → First Event | 0.12s | 0.13s | — | — | — | — |
| First Event → First Tool Call | 3.91s | 5.76s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 1 | 7 | +600.0% | — | — | — |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 1 | 7 | +600.0% | — | — | — |
| Output Tokens | 504 | 1319 | +161.7% | — | — | — |
| Reasoning / Thinking Tokens | 180 | 625 | +247.2% | — | — | — |
| Total Input Tokens | 75998 | 205404 | +170.3% | — | — | — |
| Cached Input Tokens | 66560 | 186112 | +179.6% | — | — | — |
| Uncached Input Tokens | 9438 | 19292 | +104.4% | — | — | — |
| Cost | 0.0488 | 0.1191 | +144.0% | — | — | — |
| Cached vs Uncached Token Ratio | 7.05:1 | 9.65:1 | +36.8% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | — | ✓ ✓ |
| 3 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
| 7 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 16.58s | 28.78s | +73.6% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.10s | 5.77s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 4 | +100.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 0 | -100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 4 | +100.0% | — | — | — |
| Output Tokens | 497 | 435 | -12.5% | — | — | — |
| Reasoning / Thinking Tokens | 212 | 0 | -100.0% | — | — | — |
| Total Input Tokens | 77073 | 129629 | +68.2% | — | — | — |
| Cached Input Tokens | 67584 | 115200 | +70.5% | — | — | — |
| Uncached Input Tokens | 9489 | 14429 | +52.1% | — | — | — |
| Cost | 0.0495 | 0.0703 | +42.1% | — | — | — |
| Cached vs Uncached Token Ratio | 7.12:1 | 7.98:1 | +12.1% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 2 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 3 | — | semedit/semantic_organize_imports (✓ ✓) |
| 4 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 21.28s | 40.68s | +91.2% | — | — | — |
| Process Start → First Event | 0.14s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 5.54s | 5.24s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 416 | 842 | +102.4% | — | — | — |
| Reasoning / Thinking Tokens | 110 | 215 | +95.5% | — | — | — |
| Total Input Tokens | 77023 | 160110 | +107.9% | — | — | — |
| Cached Input Tokens | 67584 | 143616 | +112.5% | — | — | — |
| Uncached Input Tokens | 9439 | 16494 | +74.7% | — | — | — |
| Cost | 0.0471 | 0.0904 | +92.0% | — | — | — |
| Cached vs Uncached Token Ratio | 7.16:1 | 8.71:1 | +21.6% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ? ✗ |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | ? ✗ |
| 8 | — | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 16.55s | 57.28s | +246.1% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.38s | 5.70s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 7 | +133.3% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 7 | +133.3% | — | — | — |
| Output Tokens | 479 | 549 | +14.6% | — | — | — |
| Reasoning / Thinking Tokens | 134 | 42 | -68.7% | — | — | — |
| Total Input Tokens | 95918 | 156319 | +63.0% | — | — | — |
| Cached Input Tokens | 84736 | 141568 | +67.1% | — | — | — |
| Uncached Input Tokens | 11182 | 14751 | +31.9% | — | — | — |
| Cost | 0.0568 | 0.0797 | +40.2% | — | — | — |
| Cached vs Uncached Token Ratio | 7.58:1 | 9.60:1 | +26.6% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ✓ ✓ |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 17.86s | 29.85s | +67.2% | — | — | — |
| Process Start → First Event | 0.12s | 0.13s | — | — | — | — |
| First Event → First Tool Call | 5.49s | 5.17s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 5 | +66.7% | — | — | — |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 5 | +66.7% | — | — | — |
| Output Tokens | 559 | 638 | +14.1% | — | — | — |
| Reasoning / Thinking Tokens | 0 | 257 | — | — | — | — |
| Total Input Tokens | 76314 | 128477 | +68.4% | — | — | — |
| Cached Input Tokens | 66560 | 114176 | +71.5% | — | — | — |
| Uncached Input Tokens | 9754 | 14301 | +46.6% | — | — | — |
| Cost | 0.0480 | 0.0755 | +57.2% | — | — | — |
| Cached vs Uncached Token Ratio | 6.82:1 | 7.98:1 | +17.0% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 22.03s | 34.09s | +54.7% | — | — | — |
| Process Start → First Event | 0.13s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 5.51s | 5.45s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 5 | +25.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 5 | +25.0% | — | — | — |
| Output Tokens | 737 | 728 | -1.2% | — | — | — |
| Reasoning / Thinking Tokens | 283 | 313 | +10.6% | — | — | — |
| Total Input Tokens | 95954 | 151546 | +57.9% | — | — | — |
| Cached Input Tokens | 85760 | 136448 | +59.1% | — | — | — |
| Uncached Input Tokens | 10194 | 15098 | +48.1% | — | — | — |
| Cost | 0.0597 | 0.0849 | +42.2% | — | — | — |
| Cached vs Uncached Token Ratio | 8.41:1 | 9.04:1 | +7.4% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 22.95s | 61.24s | +166.8% | — | — | — |
| Process Start → First Event | 0.14s | 0.16s | — | — | — | — |
| First Event → First Tool Call | 4.47s | 5.02s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 4 | +100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 754 | 939 | +24.5% | — | — | — |
| Reasoning / Thinking Tokens | 208 | 342 | +64.4% | — | — | — |
| Total Input Tokens | 96133 | 199709 | +107.7% | — | — | — |
| Cached Input Tokens | 85760 | 182016 | +112.2% | — | — | — |
| Uncached Input Tokens | 10373 | 17693 | +70.6% | — | — | — |
| Cost | 0.0594 | 0.1057 | +78.0% | — | — | — |
| Cached vs Uncached Token Ratio | 8.27:1 | 10.29:1 | +24.4% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_replace_body (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 16.22s | 48.75s | +200.5% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 4.56s | 5.46s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 4 | +33.3% | — | — | — |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 4 | +33.3% | — | — | — |
| Output Tokens | 382 | 612 | +60.2% | — | — | — |
| Reasoning / Thinking Tokens | 73 | 109 | +49.3% | — | — | — |
| Total Input Tokens | 76425 | 141929 | +85.7% | — | — | — |
| Cached Input Tokens | 66560 | 121344 | +82.3% | — | — | — |
| Uncached Input Tokens | 9865 | 20585 | +108.7% | — | — | — |
| Cost | 0.0470 | 0.0908 | +93.3% | — | — | — |
| Cached vs Uncached Token Ratio | 6.75:1 | 5.89:1 | -12.6% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | FAIL | — | — | — | — |
| Oracle L3: Clean Build | PASS | FAIL | — | — | — | — |
| Oracle L4: Verification Test | PASS | FAIL | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 19.22s | 28.51s | +48.3% | — | — | — |
| Process Start → First Event | 0.13s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 6.64s | 6.37s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 4 | 4 | 0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 2 | 0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 4 | 4 | 0% | — | — | — |
| Output Tokens | 624 | 573 | -8.2% | — | — | — |
| Reasoning / Thinking Tokens | 0 | 177 | — | — | — | — |
| Total Input Tokens | 135289 | 166691 | +23.2% | — | — | — |
| Cached Input Tokens | 120320 | 144896 | +20.4% | — | — | — |
| Uncached Input Tokens | 14969 | 21795 | +45.6% | — | — | — |
| Cost | 0.0753 | 0.1001 | +32.9% | — | — | — |
| Cached vs Uncached Token Ratio | 8.04:1 | 6.65:1 | -17.3% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 28.04s | 80.03s | +185.4% | — | — | — |
| Process Start → First Event | 0.13s | 0.14s | — | — | — | — |
| First Event → First Tool Call | 11.15s | 9.82s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 836 | 2124 | +154.1% | — | — | — |
| Reasoning / Thinking Tokens | 467 | 1223 | +161.9% | — | — | — |
| Total Input Tokens | 193285 | 452276 | +134.0% | — | — | — |
| Cached Input Tokens | 171264 | 416512 | +143.2% | — | — | — |
| Uncached Input Tokens | 22021 | 35764 | +62.4% | — | — | — |
| Cost | 0.1142 | 0.2354 | +106.2% | — | — | — |
| Cached vs Uncached Token Ratio | 7.78:1 | 11.65:1 | +49.7% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | ✓ ✓ |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
| 7 | — | ? ✗ |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 27.33s | 33.20s | +21.5% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 9.58s | 7.09s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 5 | +66.7% | — | — | — |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 5 | +66.7% | — | — | — |
| Output Tokens | 835 | 802 | -4.0% | — | — | — |
| Reasoning / Thinking Tokens | 368 | 292 | -20.7% | — | — | — |
| Total Input Tokens | 193633 | 211441 | +9.2% | — | — | — |
| Cached Input Tokens | 167168 | 185600 | +11.0% | — | — | — |
| Uncached Input Tokens | 26465 | 25841 | -2.4% | — | — | — |
| Cost | 0.1230 | 0.1247 | +1.4% | — | — | — |
| Cached vs Uncached Token Ratio | 6.32:1 | 7.18:1 | +13.7% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | FAIL | — | — | — | — |
| Oracle L3: Clean Build | PASS | FAIL | — | — | — | — |
| Oracle L4: Verification Test | PASS | FAIL | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 25.76s | 69.06s | +168.1% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 8.56s | 8.07s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 810 | 1901 | +134.7% | — | — | — |
| Reasoning / Thinking Tokens | 262 | 976 | +272.5% | — | — | — |
| Total Input Tokens | 119120 | 291380 | +144.6% | — | — | — |
| Cached Input Tokens | 107008 | 264704 | +147.4% | — | — | — |
| Uncached Input Tokens | 12112 | 26676 | +120.2% | — | — | — |
| Cost | 0.0704 | 0.1688 | +139.7% | — | — | — |
| Cached vs Uncached Token Ratio | 8.83:1 | 9.92:1 | +12.3% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_organize_imports (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 16.41s | 66.36s | +304.4% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 6.10s | 6.11s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 9 | +350.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 9 | +350.0% | — | — | — |
| Output Tokens | 518 | 1718 | +231.7% | — | — | — |
| Reasoning / Thinking Tokens | 204 | 619 | +203.4% | — | — | — |
| Total Input Tokens | 98432 | 220436 | +123.9% | — | — | — |
| Cached Input Tokens | 87808 | 196608 | +123.9% | — | — | — |
| Uncached Input Tokens | 10624 | 23828 | +124.3% | — | — | — |
| Cost | 0.0575 | 0.1379 | +139.7% | — | — | — |
| Cached vs Uncached Token Ratio | 8.27:1 | 8.25:1 | -0.2% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 3 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 23.96s | 75.30s | +214.3% | — | — | — |
| Process Start → First Event | 0.14s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 8.82s | 10.12s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 634 | 2058 | +224.6% | — | — | — |
| Reasoning / Thinking Tokens | 250 | 1144 | +357.6% | — | — | — |
| Total Input Tokens | 194621 | 376824 | +93.6% | — | — | — |
| Cached Input Tokens | 173312 | 348928 | +101.3% | — | — | — |
| Uncached Input Tokens | 21309 | 27896 | +30.9% | — | — | — |
| Cost | 0.1077 | 0.1970 | +83.0% | — | — | — |
| Cached vs Uncached Token Ratio | 8.13:1 | 12.51:1 | +53.8% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | ✓ ✓ |
| 5 | — | ✓ ✓ |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 20.07s | 50.64s | +152.3% | — | — | — |
| Process Start → First Event | 0.12s | 0.13s | — | — | — | — |
| First Event → First Tool Call | 8.62s | 5.08s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 1 | 7 | +600.0% | — | — | — |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 1 | 7 | +600.0% | — | — | — |
| Output Tokens | 605 | 1191 | +96.9% | — | — | — |
| Reasoning / Thinking Tokens | 226 | 455 | +101.3% | — | — | — |
| Total Input Tokens | 104465 | 228646 | +118.9% | — | — | — |
| Cached Input Tokens | 84992 | 205824 | +142.2% | — | — | — |
| Uncached Input Tokens | 19473 | 22822 | +17.2% | — | — | — |
| Cost | 0.0803 | 0.1291 | +60.7% | — | — | — |
| Cached vs Uncached Token Ratio | 4.36:1 | 9.02:1 | +106.6% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | — | ✓ ✓ |
| 3 | — | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_replace_body (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/report_feedback (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 22.04s | 56.11s | +154.5% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 9.41s | 9.06s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 2 | 8 | +300.0% | — | — | — |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 2 | 8 | +300.0% | — | — | — |
| Output Tokens | 663 | 999 | +50.7% | — | — | — |
| Reasoning / Thinking Tokens | 248 | 393 | +58.5% | — | — | — |
| Total Input Tokens | 162919 | 229466 | +40.8% | — | — | — |
| Cached Input Tokens | 142848 | 205824 | +44.1% | — | — | — |
| Uncached Input Tokens | 20071 | 23642 | +17.8% | — | — | — |
| Cost | 0.0973 | 0.1280 | +31.5% | — | — | — |
| Cached vs Uncached Token Ratio | 7.12:1 | 8.71:1 | +22.3% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_replace_body (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 22.64s | 47.01s | +107.6% | — | — | — |
| Process Start → First Event | 0.14s | 0.13s | — | — | — | — |
| First Event → First Tool Call | 9.42s | 5.14s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 9 | +200.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 9 | +200.0% | — | — | — |
| Output Tokens | 651 | 897 | +37.8% | — | — | — |
| Reasoning / Thinking Tokens | 234 | 237 | +1.3% | — | — | — |
| Total Input Tokens | 196350 | 169412 | -13.7% | — | — | — |
| Cached Input Tokens | 174336 | 146944 | -15.7% | — | — | — |
| Uncached Input Tokens | 22014 | 22468 | +2.1% | — | — | — |
| Cost | 0.1097 | 0.1071 | -2.4% | — | — | — |
| Cached vs Uncached Token Ratio | 7.92:1 | 6.54:1 | -17.4% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 17 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): unknown param "path"; supported keys are [bash-bin, bash_bin, file, ghc-bin, ghc-version, ghc_bin, ghc_version, haskell-standalone, hls-bin, hls-version, hls_bin, hls_version, import-maven, import_maven, java-bin, java-version, java_bin, java_version, jdtls-home, jdtls_home, kotlin-bin, kotlin_bin, language, make-bin, make_bin, metals-bin, metals-home, metals_bin, metals_home, standalone_haskell, symbol, trust-workspace, trust_workspace]: invalid operation parameters |
| 4 | — | semedit/report_feedback (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-07-generate-template-main |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/scripts/generate_template_main.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package main
func main() {
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 21.66s | 38.52s | +77.8% | — | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | — | — | — |
| First Event → First Tool Call | 7.30s | 5.75s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | — | — | — |
| Internal Tool Cycles | 3 | 6 | +100.0% | — | — | — |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 3 | 6 | +100.0% | — | — | — |
| Output Tokens | 640 | 860 | +34.4% | — | — | — |
| Reasoning / Thinking Tokens | 284 | 357 | +25.7% | — | — | — |
| Total Input Tokens | 119725 | 184509 | +54.1% | — | — | — |
| Cached Input Tokens | 99840 | 149760 | +50.0% | — | — | — |
| Uncached Input Tokens | 19885 | 34749 | +74.7% | — | — | — |
| Cost | 0.0862 | 0.1395 | +61.8% | — | — | — |
| Cached vs Uncached Token Ratio | 5.02:1 | 4.31:1 | -14.2% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | PASS | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 4 | — | semedit/semantic_replace_body (✓ ✓) |
| 5 | — | ✓ ✓ |
| 6 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
Test case: task-09-composite-refactor
Target: codex/gpt-6-luna/medium
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 26.07s | 67.17s | +157.6% | 20.77s | 101.16s | +387.2% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 5.28s | 5.10s | — | 5.77s | 7.13s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 11 | +83.3% | 3 | 17 | +466.7% |
| Initial Load / Discovery Turns | 4 | 1 | -75.0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 11 | +83.3% | 3 | 17 | +466.7% |
| Output Tokens | 901 | 1738 | +92.9% | 701 | 2699 | +285.0% |
| Reasoning / Thinking Tokens | 224 | 1011 | +351.3% | 84 | 1489 | +1672.6% |
| Total Input Tokens | 138787 | 309077 | +122.7% | 76559 | 357716 | +367.2% |
| Cached Input Tokens | 126208 | 284672 | +125.6% | 66560 | 330496 | +396.5% |
| Uncached Input Tokens | 12579 | 24405 | +94.0% | 9999 | 27220 | +172.2% |
| Cost | 0.0771 | 0.1665 | +116.1% | 0.0515 | 0.2030 | +294.6% |
| Cached vs Uncached Token Ratio | 10.03:1 | 11.66:1 | +16.3% | 6.66:1 | 12.14:1 | +82.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 4 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | ✓ ✓ |
| 8 | — | semedit/semantic_rename (✓ ✓) |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | ? ✗ |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | — | semedit/semantic_outline (✓ ✓) |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | semedit/semantic_find_references (✓ ✗): load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463375966287000/config/config.go:4:2: "archive/tar" imported and not used |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_find_references (✓ ✗): load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463375966287000/config/config_test.go:6:7: undefined: DefaultConfig |
| 10 | — | semedit/semantic_lookup (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_rename (✓ ✓) |
| 14 | — | semedit/semantic_insert_function (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 20.29s | 52.47s | +158.6% | 22.32s | 51.99s | +132.9% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 4.60s | 4.58s | — | 4.38s | 4.52s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 4 | 10 | +150.0% | 5 | 11 | +120.0% |
| Initial Load / Discovery Turns | 4 | 1 | -75.0% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 4 | 10 | +150.0% | 5 | 11 | +120.0% |
| Output Tokens | 490 | 1099 | +124.3% | 706 | 933 | +32.2% |
| Reasoning / Thinking Tokens | 119 | 266 | +123.5% | 161 | 162 | +0.6% |
| Total Input Tokens | 97594 | 269332 | +176.0% | 117570 | 307220 | +161.3% |
| Cached Input Tokens | 86784 | 249856 | +187.9% | 105984 | 282880 | +166.9% |
| Uncached Input Tokens | 10810 | 19476 | +80.2% | 11586 | 24340 | +110.1% |
| Cost | 0.0563 | 0.1282 | +127.6% | 0.0663 | 0.1453 | +119.1% |
| Cached vs Uncached Token Ratio | 8.03:1 | 12.83:1 | +59.8% | 9.15:1 | 11.62:1 | +27.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | ? ✗ |
| 5 | — | semedit/semantic_rename (✓ ✓) |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ? ✗ |
| 9 | — | ✓ ✓ |
| 10 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_rename (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 27.38s | 60.55s | +121.1% | 24.08s | 76.00s | +215.6% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 7.10s | 5.16s | — | 5.06s | 5.55s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 9 | +50.0% | 5 | 16 | +220.0% |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 9 | +50.0% | 5 | 16 | +220.0% |
| Output Tokens | 805 | 813 | +1.0% | 813 | 1874 | +130.5% |
| Reasoning / Thinking Tokens | 235 | 278 | +18.3% | 81 | 729 | +800.0% |
| Total Input Tokens | 135848 | 177196 | +30.4% | 116092 | 302104 | +160.2% |
| Cached Input Tokens | 103936 | 150528 | +44.8% | 104960 | 277760 | +164.6% |
| Uncached Input Tokens | 31912 | 26668 | -16.4% | 11132 | 24344 | +118.7% |
| Cost | 0.1188 | 0.1179 | -0.7% | 0.0652 | 0.1628 | +149.6% |
| Cached vs Uncached Token Ratio | 3.26:1 | 5.64:1 | +73.3% | 9.43:1 | 11.41:1 | +21.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_verify (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Load in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463295611124000/config/config.go: symbol not found |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 7 | — | ? ✗ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_rename (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 25.65s | 57.74s | +125.1% | 22.50s | 61.43s | +173.1% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 5.12s | 4.54s | — | 5.05s | 5.38s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 10 | +100.0% | 4 | 11 | +175.0% |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | 3 | 1 | -66.7% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 10 | +100.0% | 4 | 11 | +175.0% |
| Output Tokens | 936 | 1205 | +28.7% | 756 | 921 | +21.8% |
| Reasoning / Thinking Tokens | 110 | 578 | +425.5% | 119 | 284 | +138.7% |
| Total Input Tokens | 117121 | 227090 | +93.9% | 95974 | 253817 | +164.5% |
| Cached Input Tokens | 104960 | 204544 | +94.9% | 85760 | 228096 | +166.0% |
| Uncached Input Tokens | 12161 | 22546 | +85.4% | 10214 | 25721 | +151.8% |
| Cost | 0.0697 | 0.1298 | +86.2% | 0.0579 | 0.1364 | +135.5% |
| Cached vs Uncached Token Ratio | 8.63:1 | 9.07:1 | +5.1% | 8.40:1 | 8.87:1 | +5.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | — | semedit/semantic_rename (✓ ✓) |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 26.38s | 69.57s | +163.7% | 19.23s | 53.76s | +179.6% |
| Process Start → First Event | 0.14s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 5.13s | 4.44s | — | 4.42s | 4.34s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 12 | +140.0% | 3 | 11 | +266.7% |
| Initial Load / Discovery Turns | 5 | 2 | -60.0% | 3 | 5 | +66.7% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 12 | +140.0% | 3 | 11 | +266.7% |
| Output Tokens | 880 | 1399 | +59.0% | 530 | 1080 | +103.8% |
| Reasoning / Thinking Tokens | 218 | 580 | +166.1% | 73 | 402 | +450.7% |
| Total Input Tokens | 116877 | 299164 | +156.0% | 97820 | 284047 | +190.4% |
| Cached Input Tokens | 104960 | 277504 | +164.4% | 87808 | 262144 | +198.5% |
| Uncached Input Tokens | 11917 | 21660 | +81.8% | 10012 | 21903 | +118.8% |
| Cost | 0.0698 | 0.1483 | +112.5% | 0.0545 | 0.1388 | +154.6% |
| Cached vs Uncached Token Ratio | 8.81:1 | 12.81:1 | +45.5% | 8.77:1 | 11.97:1 | +36.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | — | ? ✗ |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_rename (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_organize_imports (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | ✓ ✓ |
| 4 | — | ✓ ✓ |
| 5 | — | ✓ ✓ |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_rename (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 20.11s | 59.81s | +197.5% | 27.91s | 84.37s | +202.3% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.14s | 0.12s | — |
| First Event → First Tool Call | 5.34s | 5.73s | — | 4.65s | 5.63s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 3 | 11 | +266.7% | 6 | 16 | +166.7% |
| Initial Load / Discovery Turns | 2 | 1 | -50.0% | 6 | 2 | -66.7% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 3 | 11 | +266.7% | 6 | 16 | +166.7% |
| Output Tokens | 533 | 1357 | +154.6% | 736 | 1999 | +171.6% |
| Reasoning / Thinking Tokens | 44 | 385 | +775.0% | 172 | 922 | +436.0% |
| Total Input Tokens | 77278 | 303267 | +292.4% | 157349 | 271691 | +72.7% |
| Cached Input Tokens | 66560 | 280576 | +321.5% | 144384 | 249088 | +72.5% |
| Uncached Input Tokens | 10718 | 22691 | +111.7% | 12965 | 22603 | +74.3% |
| Cost | 0.0506 | 0.1486 | +193.5% | 0.0799 | 0.1553 | +94.5% |
| Cached vs Uncached Token Ratio | 6.21:1 | 12.37:1 | +99.1% | 11.14:1 | 11.02:1 | -1.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | — | ✓ ✓ |
| 5 | — | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_rename", "status": "error", "error": "invalid arguments: unexpected end of JSON input" } ], "final_diff": "" } |
| 6 | — | semedit/semantic_rename (✓ ✓) |
| 7 | — | semedit/semantic_insert_function (✓ ✓) |
| 8 | — | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): param "symbol" is required: invalid operation parameters |
| 7 | — | ✓ ✓ |
| 8 | — | ? ✗ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✗): resolve Load in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463265524534000/config/config.go: symbol not found |
| 10 | — | semedit/semantic_lookup (✓ ✗): resolve Load in config/config.go: symbol not found |
| 11 | — | semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: The workspace-wide semantic rename may modify test files, which the user explicitly forbade; use a narrowly scoped production-file edit instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_organize_imports (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 30.31s | 45.93s | +51.5% | 21.09s | 78.47s | +272.0% |
| Process Start → First Event | 0.12s | 0.14s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 4.98s | 4.51s | — | 5.08s | 4.59s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 6 | 7 | +16.7% | 3 | 10 | +233.3% |
| Initial Load / Discovery Turns | 4 | 1 | -75.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 7 | +16.7% | 3 | 10 | +233.3% |
| Output Tokens | 1084 | 670 | -38.2% | 715 | 1077 | +50.6% |
| Reasoning / Thinking Tokens | 242 | 192 | -20.7% | 147 | 374 | +154.4% |
| Total Input Tokens | 138772 | 193399 | +39.4% | 96835 | 215654 | +122.7% |
| Cached Input Tokens | 126208 | 174080 | +37.9% | 85760 | 197376 | +130.1% |
| Uncached Input Tokens | 12564 | 19319 | +53.8% | 11075 | 18278 | +65.0% |
| Cost | 0.0795 | 0.1026 | +29.0% | 0.0599 | 0.1132 | +88.9% |
| Cached vs Uncached Token Ratio | 10.05:1 | 9.01:1 | -10.3% | 7.74:1 | 10.80:1 | +39.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | semedit/semantic_organize_imports (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | — | semedit/semantic_lookup (✓ ✗): resolve DefaultConfig: symbol not found |
| 5 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | — | ✓ ✓ |
| 7 | — | semedit/semantic_rename (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_organize_imports (✓ ✓) |
| 10 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 33.54s | 46.96s | +40.0% | 24.97s | 85.02s | +240.5% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 7.61s | 5.47s | — | 5.15s | 8.20s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 11 | +57.1% | 4 | 15 | +275.0% |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 11 | +57.1% | 4 | 15 | +275.0% |
| Output Tokens | 982 | 1042 | +6.1% | 881 | 2089 | +137.1% |
| Reasoning / Thinking Tokens | 251 | 304 | +21.1% | 100 | 1232 | +1132.0% |
| Total Input Tokens | 157952 | 282824 | +79.1% | 97361 | 430845 | +342.5% |
| Cached Input Tokens | 144384 | 258560 | +79.1% | 86784 | 397056 | +357.5% |
| Uncached Input Tokens | 13568 | 24264 | +78.8% | 10577 | 33789 | +219.5% |
| Cost | 0.0854 | 0.1421 | +66.4% | 0.0604 | 0.2252 | +272.9% |
| Cached vs Uncached Token Ratio | 10.64:1 | 10.66:1 | +0.1% | 8.20:1 | 11.75:1 | +43.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ? ✗ |
| 10 | — | semedit/semantic_verify (✓ ✓) |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | — | ✓ ✓ |
| 6 | — | semedit/semantic_find_references (✓ ✗): load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463399078155000/config/config.go:4:2: "archive/tar" imported and not used |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | ? ✗ |
| 10 | — | semedit/semantic_organize_imports (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_insert_function (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 24.06s | 62.39s | +159.3% | 23.11s | 137.10s | +493.2% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.12s | 0.13s | — |
| First Event → First Tool Call | 4.11s | 4.78s | — | 4.61s | 4.31s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 5 | +400.0% |
| Internal Tool Cycles | 6 | 11 | +83.3% | 6 | 21 | +250.0% |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | 4 | 4 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 11 | +83.3% | 6 | 21 | +250.0% |
| Output Tokens | 782 | 1002 | +28.1% | 667 | 7844 | +1076.0% |
| Reasoning / Thinking Tokens | 205 | 346 | +68.8% | 109 | 2727 | +2401.8% |
| Total Input Tokens | 137722 | 279166 | +102.7% | 136362 | 2296226 | +1583.9% |
| Cached Input Tokens | 118016 | 254208 | +115.4% | 123136 | 2141696 | +1639.3% |
| Uncached Input Tokens | 19706 | 24958 | +26.7% | 13226 | 154530 | +1068.4% |
| Cost | 0.0911 | 0.1428 | +56.7% | 0.0735 | 1.0539 | +1332.9% |
| Cached vs Uncached Token Ratio | 5.99:1 | 10.19:1 | +70.1% | 9.31:1 | 13.86:1 | +48.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | FAIL | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | FAIL | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_rename (✓ ✓) |
| 8 | — | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename may modify test files, which the user explicitly forbade; the requested source rename does not authorize that side effect. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, cla |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve DefaultConfig in config/config.go: symbol not found |
| 7 | — | semedit/semantic_outline (✓ ✓) |
| 8 | — | semedit/semantic_find_references (✓ ✗): load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790463416179444000/config/config.go:4:2: "archive/tar" imported and not used |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_insert_function (✓ ✓) |
| 11 | — | semedit/semantic_organize_imports (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | ? ✗ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-09-composite-refactor |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_09_composite_refactor.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package config
import (
"archive/tar"
"fmt"
)
type Config struct {
Host string
Port int
}
func (c *Config) Addr() string {
return fmt.Sprintf("%s:%d", c.Host, c.Port)
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 27.11s | 158.12s | +483.2% | 25.64s | 43.72s | +70.5% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 4.21s | 5.49s | — | 5.11s | 7.67s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 5 | +400.0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 22 | +340.0% | 5 | 8 | +60.0% |
| Initial Load / Discovery Turns | 5 | 4 | -20.0% | 3 | 1 | -66.7% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 22 | +340.0% | 5 | 8 | +60.0% |
| Output Tokens | 959 | 16696 | +1641.0% | 952 | 601 | -36.9% |
| Reasoning / Thinking Tokens | 380 | 10591 | +2687.1% | 135 | 121 | -10.4% |
| Total Input Tokens | 139200 | 2252355 | +1518.1% | 117615 | 212631 | +80.8% |
| Cached Input Tokens | 127232 | 2091520 | +1543.9% | 105984 | 194304 | +83.3% |
| Uncached Input Tokens | 11968 | 160835 | +1243.9% | 11631 | 18327 | +57.6% |
| Cost | 0.0785 | 1.2661 | +1513.5% | 0.0692 | 0.1034 | +49.5% |
| Cached vs Uncached Token Ratio | 10.63:1 | 13.00:1 | +22.3% | 9.11:1 | 10.60:1 | +16.4% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | FAIL | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | FAIL | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | FAIL | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve DefaultConfig in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-09-composite-refactor_1790463233001425000/config/config.go: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | — | semedit/semantic_lookup (✓ ✗): resolve Port in config/config.go: symbol not found |
| 7 | — | ? ✗ |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename may modify test files, which the user explicitly forbade; the planned scope is not limited to config/config.go. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why it |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | ? ✗ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_outline (✓ ✓) |
| 21 | — | ✓ ✓ |
| 22 | — | ? ✗ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 3 | ? ✗ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | — | semedit/semantic_insert_function (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
Test case: task-11-mixed-sink-api-migration
Target: codex/gpt-6-luna/medium
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 37.37s | 163.40s | +337.2% | 248.61s | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | 0.23s | — | — |
| First Event → First Tool Call | 3.86s | 5.81s | — | 5.22s | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 5 | +400.0% | 5 | — | — |
| Internal Tool Cycles | 6 | 25 | +316.7% | 14 | — | — |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 2 | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | — | — |
| Tool Invocations (per run) | 6 | 25 | +316.7% | 14 | — | — |
| Output Tokens | 1326 | 11620 | +776.3% | 18185 | — | — |
| Reasoning / Thinking Tokens | 592 | 3921 | +562.3% | 7748 | — | — |
| Total Input Tokens | 162124 | 2369906 | +1361.8% | 1677358 | — | — |
| Cached Input Tokens | 141312 | 2184192 | +1445.7% | 1562368 | — | — |
| Uncached Input Tokens | 20812 | 185714 | +792.3% | 114990 | — | — |
| Cost | 0.1113 | 1.2046 | +982.0% | 1.0022 | — | — |
| Cached vs Uncached Token Ratio | 6.79:1 | 11.76:1 | +73.2% | 13.59:1 | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | — | — |
| Oracle L4: Verification Test | PASS | FAIL | — | PASS | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | ✓ ✓ |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Verification on path '.' may format and write across the entire workspace, including protected tests, despite the user’s explicit prohibition on test edits; use a narrowly scoped read-only check instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for con |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_insert_function (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | not published |
| 2 | ✓ ✓ | not published |
| 3 | ✓ ✓ | not published |
| 4 | ✓ ✓ | not published |
| 5 | ? ✗ | not published |
| 6 | ✓ ✓ | not published |
| 7 | ✓ ✓ | not published |
| 8 | ✓ ✓ | not published |
| 9 | ✓ ✓ | not published |
| 10 | ✓ ✓ | not published |
| 11 | ✓ ✓ | not published |
| 12 | ✓ ✓ | not published |
| 13 | ✓ ✓ | not published |
| 14 | ✓ ✓ | not published |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 107.96s | 162.28s | +50.3% | 62.86s | 267.56s | +325.6% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.37s | 4.53s | — | 5.47s | 4.40s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 1 | 5 | +400.0% |
| Internal Tool Cycles | 14 | 21 | +50.0% | 9 | 41 | +355.6% |
| Initial Load / Discovery Turns | 2 | 3 | +50.0% | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 14 | 21 | +50.0% | 9 | 41 | +355.6% |
| Output Tokens | 13170 | 11645 | -11.6% | 1971 | 20964 | +963.6% |
| Reasoning / Thinking Tokens | 3749 | 5104 | +36.1% | 510 | 10234 | +1906.7% |
| Total Input Tokens | 1442295 | 2351515 | +63.0% | 218986 | 4919381 | +2146.4% |
| Cached Input Tokens | 1309184 | 2205696 | +68.5% | 198144 | 4662528 | +2253.1% |
| Uncached Input Tokens | 133111 | 145819 | +9.5% | 20842 | 256853 | +1132.4% |
| Cost | 0.8716 | 1.1253 | +29.1% | 0.1327 | 2.1977 | +1556.8% |
| Cached vs Uncached Token Ratio | 9.84:1 | 15.13:1 | +53.8% | 9.51:1 | 18.15:1 | +90.9% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | PASS | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 14 | ✓ ✓ | ✓ ✓ |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 21 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✗): param "symbol" is required: invalid operation parameters |
| 14 | — | semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations |
| 15 | — | ? ✗ |
| 16 | — | semedit/semantic_insert_function (✓ ✓) |
| 17 | — | semedit/semantic_insert_function (✓ ✓) |
| 18 | — | semedit/semantic_insert_function (✓ ✓) |
| 19 | — | semedit/semantic_replace_body (✓ ✓) |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | semedit/semantic_replace_body (✓ ✓) |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ? ✗ |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_outline (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_outline (✓ ✓) |
| 32 | — | semedit/semantic_verify (✓ ✓) |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | semedit/semantic_verify (✓ ✓) |
| 35 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 36 | — | semedit/semantic_verify (✓ ✓) |
| 37 | — | semedit/semantic_replace_body (✓ ✓) |
| 38 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 39 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 40 | — | ✓ ✓ |
| 41 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 60.95s | 80.94s | +32.8% | 136.36s | 218.37s | +60.1% |
| Process Start → First Event | 0.13s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 4.64s | 4.75s | — | 4.40s | 4.98s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 6 | 11 | +83.3% | 14 | 47 | +235.7% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 4 | 3 | -25.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 11 | +83.3% | 14 | 47 | +235.7% |
| Output Tokens | 2365 | 1845 | -22.0% | 18049 | 17805 | -1.4% |
| Reasoning / Thinking Tokens | 795 | 737 | -7.3% | 9065 | 8044 | -11.3% |
| Total Input Tokens | 145381 | 309664 | +113.0% | 2584331 | 3058666 | +18.4% |
| Cached Input Tokens | 121088 | 285440 | +135.7% | 2416896 | 2836480 | +17.4% |
| Uncached Input Tokens | 24293 | 24224 | -0.3% | 167435 | 222186 | +32.7% |
| Cost | 0.1305 | 0.1642 | +25.8% | 1.3617 | 1.5877 | +16.6% |
| Cached vs Uncached Token Ratio | 4.98:1 | 11.78:1 | +136.4% | 14.43:1 | 12.77:1 | -11.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | FAIL | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | — | semedit/semantic_replace_body (✓ ✓) |
| 8 | — | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_verify (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ? ✗ |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_outline (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
| 28 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 29 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_verify (✓ ✓) |
| 32 | — | ✓ ✓ |
| 33 | — | ✓ ✓ |
| 34 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 35 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 36 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 37 | — | ✓ ✓ |
| 38 | — | semedit/semantic_verify (✓ ✓) |
| 39 | — | ✓ ✓ |
| 40 | — | ✓ ✓ |
| 41 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 42 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 43 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 44 | — | ✓ ✓ |
| 45 | — | semedit/semantic_replace_body (✓ ✓) |
| 46 | — | ✓ ✓ |
| 47 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 109.72s | 259.24s | +136.3% | 143.00s | 228.09s | +59.5% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.01s | 4.54s | — | 6.37s | 4.99s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 10 | 35 | +250.0% | 13 | 40 | +207.7% |
| Initial Load / Discovery Turns | 1 | 1 | 0% | 3 | 4 | +33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 10 | 35 | +250.0% | 13 | 40 | +207.7% |
| Output Tokens | 12430 | 18026 | +45.0% | 20146 | 17690 | -12.2% |
| Reasoning / Thinking Tokens | 4593 | 7272 | +58.3% | 10372 | 8190 | -21.0% |
| Total Input Tokens | 1611452 | 3971751 | +146.5% | 1862526 | 3514762 | +88.7% |
| Cached Input Tokens | 1509888 | 3785984 | +150.7% | 1739520 | 3288576 | +89.1% |
| Uncached Input Tokens | 101564 | 185767 | +82.9% | 123006 | 226186 | +83.9% |
| Cost | 0.8442 | 1.7271 | +104.6% | 1.1239 | 1.7111 | +52.3% |
| Cached vs Uncached Token Ratio | 14.87:1 | 20.38:1 | +37.1% | 14.14:1 | 14.54:1 | +2.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | FAIL | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_organize_imports (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | semedit/semantic_replace_body (✓ ✓) |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ✓ ✓ |
| 28 | — | semedit/semantic_insert_function (✓ ✓) |
| 29 | — | semedit/semantic_replace_body (✓ ✓) |
| 30 | — | semedit/semantic_replace_body (✓ ✓) |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_organize_imports (✓ ✓) |
| 33 | — | semedit/semantic_replace_body (✓ ✓) |
| 34 | — | semedit/semantic_verify (✓ ✓) |
| 35 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 27 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 5 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve NormalizeKind in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790465236492930000/audit/normalize.go: symbol not found |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 14 | — | semedit/semantic_insert_function (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | semedit/semantic_replace_body (✓ ✓) |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Repository-wide semantic verification may format and modify test files, conflicting with the user's explicit prohibition on editing tests; use a read-only or narrower verification instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 30 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 36 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 37 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 38 | — | semedit/semantic_replace_body (✓ ✓) |
| 39 | — | ✓ ✓ |
| 40 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 65.07s | 98.83s | +51.9% | 137.97s | 199.48s | +44.6% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 5.38s | 5.07s | — | 5.13s | 4.53s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 8 | 16 | +100.0% | 13 | 43 | +230.8% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 3 | 4 | +33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 16 | +100.0% | 13 | 43 | +230.8% |
| Output Tokens | 1842 | 1654 | -10.2% | 16205 | 15478 | -4.5% |
| Reasoning / Thinking Tokens | 665 | 651 | -2.1% | 6201 | 6239 | +0.6% |
| Total Input Tokens | 251215 | 391863 | +56.0% | 2254503 | 3129730 | +38.8% |
| Cached Input Tokens | 232448 | 358144 | +54.1% | 2013696 | 2923776 | +45.2% |
| Uncached Input Tokens | 18767 | 33719 | +79.7% | 240807 | 205954 | -14.5% |
| Cost | 0.1364 | 0.2026 | +48.6% | 1.3855 | 1.5173 | +9.5% |
| Cached vs Uncached Token Ratio | 12.39:1 | 10.62:1 | -14.2% | 8.36:1 | 14.20:1 | +69.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | FAIL | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification may format and modify protected test or module files, contrary to the user's explicit instruction not to edit tests or go.mod/go.sum; verify only the changed source paths instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without ask |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ? ✗ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | ? ✗ |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ? ✗ |
| 19 | — | ? ✗ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | ✓ ✓ |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | semedit/semantic_replace_body (✓ ✓) |
| 28 | — | semedit/semantic_verify (✓ ✓) |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | semedit/semantic_replace_body (✓ ✓) |
| 37 | — | semedit/semantic_verify (✓ ✓) |
| 38 | — | ✓ ✓ |
| 39 | — | ✓ ✓ |
| 40 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 41 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 42 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 43 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 109.62s | 196.23s | +79.0% | 170.46s | 175.35s | +2.9% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 4.58s | 6.69s | — | 4.54s | 5.26s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 2 | -60.0% |
| Internal Tool Cycles | 12 | 44 | +266.7% | 19 | 27 | +42.1% |
| Initial Load / Discovery Turns | 2 | 4 | +100.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 12 | 44 | +266.7% | 19 | 27 | +42.1% |
| Output Tokens | 13154 | 13952 | +6.1% | 20141 | 7660 | -62.0% |
| Reasoning / Thinking Tokens | 5767 | 5048 | -12.5% | 9590 | 3570 | -62.8% |
| Total Input Tokens | 1650359 | 2804497 | +69.9% | 3073510 | 1094968 | -64.4% |
| Cached Input Tokens | 1540864 | 2594048 | +68.4% | 2887680 | 1020160 | -64.7% |
| Uncached Input Tokens | 109495 | 210449 | +92.2% | 185830 | 74808 | -59.7% |
| Cost | 0.8955 | 1.4121 | +57.7% | 1.5581 | 0.5824 | -62.6% |
| Cached vs Uncached Token Ratio | 14.07:1 | 12.33:1 | -12.4% | 15.54:1 | 13.64:1 | -12.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | FAIL | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Deliver: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Normalize: symbol not found |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Probe: symbol not found |
| 7 | ? ✗ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Normalize in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790463914583233000/audit/normalize.go: symbol not found |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification runs formatting and may write files, potentially modifying the explicitly protected test files; a read-only or source-scoped verification is required. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | semedit/semantic_verify (✓ ✓) |
| 28 | — | semedit/semantic_verify (✓ ✓) |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | ✓ ✓ |
| 34 | — | semedit/semantic_verify (✓ ✓) |
| 35 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 36 | — | ✓ ✓ |
| 37 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 38 | — | ✓ ✓ |
| 39 | — | semedit/semantic_verify (✓ ✓) |
| 40 | — | ✓ ✓ |
| 41 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 42 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 43 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 44 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 15 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 16 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 17 | ✓ ✓ | ✓ ✓ |
| 18 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 19 | ✓ ✓ | ✓ ✓ |
| 20 | — | ? ✗ |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 109.54s | 295.32s | +169.6% | 152.72s | 236.72s | +55.0% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 4.43s | 4.59s | — | 4.80s | 5.49s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 11 | 39 | +254.5% | 11 | 54 | +390.9% |
| Initial Load / Discovery Turns | 3 | 3 | 0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | 39 | +254.5% | 11 | 54 | +390.9% |
| Output Tokens | 11213 | 16445 | +46.7% | 21453 | 16358 | -23.7% |
| Reasoning / Thinking Tokens | 3724 | 6705 | +80.0% | 10498 | 5941 | -43.4% |
| Total Input Tokens | 1468899 | 3618341 | +146.3% | 1704131 | 4581752 | +168.9% |
| Cached Input Tokens | 1363968 | 3419136 | +150.7% | 1576192 | 4319744 | +174.1% |
| Uncached Input Tokens | 104931 | 199205 | +89.8% | 127939 | 262008 | +104.8% |
| Cost | 0.7900 | 1.6422 | +107.9% | 1.1133 | 2.0137 | +80.9% |
| Cached vs Uncached Token Ratio | 13.00:1 | 17.16:1 | +32.0% | 12.32:1 | 16.49:1 | +33.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | PASS | — | FAIL | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification may format and write Go files, potentially modifying forbidden tests or protected files; a read-only or narrowly scoped verification is safer. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: The verification targets the entire audit directory and may format/write test files, directly conflicting with the user's explicit prohibition on editing tests; a narrower source-only check is required. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confir |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_verify (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_insert_function (✓ ✓) |
| 33 | — | semedit/semantic_replace_body (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | semedit/semantic_verify (✓ ✓) |
| 37 | — | semedit/semantic_verify (✓ ✓) |
| 38 | — | semedit/semantic_verify (✓ ✓) |
| 39 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Deliver: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✗): resolve Write in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790464800465799000/audit/transaction.go: symbol not found |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_insert_function (✓ ✓) |
| 17 | — | semedit/semantic_insert_function (✓ ✓) |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | semedit/semantic_replace_body (✓ ✓) |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | ✓ ✓ |
| 30 | — | semedit/semantic_verify (✓ ✓) |
| 31 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 32 | — | ✓ ✓ |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | ✓ ✓ |
| 36 | — | semedit/semantic_verify (✓ ✓) |
| 37 | — | ✓ ✓ |
| 38 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 39 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 40 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 41 | — | ✓ ✓ |
| 42 | — | ✓ ✓ |
| 43 | — | semedit/semantic_verify (✓ ✓) |
| 44 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 45 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 46 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 47 | — | ✓ ✓ |
| 48 | — | semedit/semantic_insert_function (✓ ✓) |
| 49 | — | semedit/semantic_replace_body (✓ ✓) |
| 50 | — | semedit/semantic_replace_body (✓ ✓) |
| 51 | — | semedit/semantic_replace_body (✓ ✓) |
| 52 | — | ✓ ✓ |
| 53 | — | ✓ ✓ |
| 54 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 106.12s | 209.94s | +97.8% | 136.56s | 122.38s | -10.4% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.70s | 5.71s | — | 4.38s | 5.90s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 3 | -40.0% |
| Internal Tool Cycles | 11 | 40 | +263.6% | 13 | 27 | +107.7% |
| Initial Load / Discovery Turns | 4 | 4 | 0% | 3 | 7 | +133.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | 40 | +263.6% | 13 | 27 | +107.7% |
| Output Tokens | 10451 | 16482 | +57.7% | 16313 | 5464 | -66.5% |
| Reasoning / Thinking Tokens | 2720 | 7226 | +165.7% | 7992 | 1739 | -78.2% |
| Total Input Tokens | 1501213 | 5799268 | +286.3% | 1861044 | 1534790 | -17.5% |
| Cached Input Tokens | 1376000 | 5308416 | +285.8% | 1745920 | 1425152 | -18.4% |
| Uncached Input Tokens | 125213 | 490852 | +292.0% | 115124 | 109638 | -4.8% |
| Cost | 0.8217 | 2.8506 | +246.9% | 1.0281 | 0.7204 | -29.9% |
| Cached vs Uncached Token Ratio | 10.99:1 | 10.81:1 | -1.6% | 15.17:1 | 13.00:1 | -14.3% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_outline (✓ ✓) |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 28 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_verify (✓ ✓) |
| 32 | — | ✓ ✓ |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | semedit/semantic_verify (✓ ✓) |
| 36 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 37 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 38 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 39 | — | ✓ ✓ |
| 40 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 20 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Normalize: symbol not found |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve BufferedSink.Write: symbol not found |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | ✓ ✓ | ? ✗ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | ✓ ✓ |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 198.57s | — | — | 129.89s | 196.16s | +51.0% |
| Process Start → First Event | 0.12s | — | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.52s | — | — | 5.20s | 5.37s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | — | — | 5 | 4 | -20.0% |
| Internal Tool Cycles | 13 | — | — | 15 | 30 | +100.0% |
| Initial Load / Discovery Turns | 2 | — | — | 3 | 3 | 0% |
| MCP Discovery / Schema Turns | 0 | — | — | 0 | 0 | — |
| Tool Invocations (per run) | 13 | — | — | 15 | 30 | +100.0% |
| Output Tokens | 12772 | — | — | 16433 | 13301 | -19.1% |
| Reasoning / Thinking Tokens | 5519 | — | — | 7458 | 5766 | -22.7% |
| Total Input Tokens | 1422298 | — | — | 2006634 | 2995271 | +49.3% |
| Cached Input Tokens | 1299456 | — | — | 1873408 | 2816512 | +50.3% |
| Uncached Input Tokens | 122842 | — | — | 133226 | 178759 | +34.2% |
| Cost | 0.8606 | — | — | 1.1001 | 1.3894 | +26.3% |
| Cached vs Uncached Token Ratio | 10.58:1 | — | — | 14.06:1 | 15.76:1 | +12.0% |
| Oracle L1: Mutation Policy | PASS | — | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | — | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | — | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | — | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | — | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | not published |
| 2 | ✓ ✓ | not published |
| 3 | ✓ ✓ | not published |
| 4 | ✓ ✓ | not published |
| 5 | ✓ ✓ | not published |
| 6 | ✓ ✓ | not published |
| 7 | ✓ ✓ | not published |
| 8 | ✓ ✓ | not published |
| 9 | ✓ ✓ | not published |
| 10 | ✓ ✓ | not published |
| 11 | ✓ ✓ | not published |
| 12 | ✓ ✓ | not published |
| 13 | ✓ ✓ | not published |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve Probe: symbol not found |
| 9 | ✓ ✓ | ? ✗ |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 27 | — | semedit/semantic_replace_body (✓ ✓) |
| 28 | — | semedit/semantic_replace_body (✓ ✓) |
| 29 | — | semedit/semantic_verify (✓ ✓) |
| 30 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 98.17s | 171.24s | +74.4% | 114.79s | 222.17s | +93.6% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.54s | — |
| First Event → First Tool Call | 11.96s | 5.11s | — | 5.65s | 4.67s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 12 | 27 | +125.0% | 14 | 35 | +150.0% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 4 | 3 | -25.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 12 | 27 | +125.0% | 14 | 35 | +150.0% |
| Output Tokens | 9895 | 13166 | +33.1% | 13163 | 13477 | +2.4% |
| Reasoning / Thinking Tokens | 3457 | 5720 | +65.5% | 5407 | 4869 | -10.0% |
| Total Input Tokens | 1739183 | 2600686 | +49.5% | 1866213 | 3741827 | +100.5% |
| Cached Input Tokens | 1630976 | 2440192 | +49.6% | 1732864 | 3456000 | +99.4% |
| Uncached Input Tokens | 108207 | 160494 | +48.3% | 133349 | 285827 | +114.3% |
| Cost | 0.8452 | 1.2474 | +47.6% | 0.9987 | 1.8079 | +81.0% |
| Cached vs Uncached Token Ratio | 15.07:1 | 15.20:1 | +0.9% | 12.99:1 | 12.09:1 | -7.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | PASS | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | ✓ ✓ |
| 23 | — | ✓ ✓ |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | ✓ ✓ |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 14 | ✓ ✓ | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_replace_body (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_organize_imports (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | ✓ ✓ |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
| 28 | — | semedit/semantic_insert_function (✓ ✓) |
| 29 | — | semedit/semantic_replace_body (✓ ✓) |
| 30 | — | semedit/semantic_replace_body (✓ ✓) |
| 31 | — | semedit/semantic_organize_imports (✓ ✓) |
| 32 | — | semedit/semantic_replace_body (✓ ✓) |
| 33 | — | semedit/semantic_verify (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 94.77s | — | — | 200.48s | 245.57s | +22.5% |
| Process Start → First Event | 0.12s | — | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 8.46s | — | — | 18.26s | 9.68s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | — | — | 5 | 5 | 0% |
| Internal Tool Cycles | 11 | — | — | 10 | 37 | +270.0% |
| Initial Load / Discovery Turns | 3 | — | — | 2 | 3 | +50.0% |
| MCP Discovery / Schema Turns | 0 | — | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | — | — | 10 | 37 | +270.0% |
| Output Tokens | 11353 | — | — | 22893 | 21144 | -7.6% |
| Reasoning / Thinking Tokens | 6018 | — | — | 8078 | 9487 | +17.4% |
| Total Input Tokens | 1377547 | — | — | 2708532 | 3756147 | +38.7% |
| Cached Input Tokens | 1281536 | — | — | 2518784 | 3509760 | +39.3% |
| Uncached Input Tokens | 96011 | — | — | 189748 | 246387 | +29.8% |
| Cost | 0.7775 | — | — | 1.4912 | 1.8763 | +25.8% |
| Cached vs Uncached Token Ratio | 13.35:1 | — | — | 13.27:1 | 14.24:1 | +7.3% |
| Oracle L1: Mutation Policy | PASS | — | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | — | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | — | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | — | — | FAIL | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | — | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | not published |
| 2 | ✓ ✓ | not published |
| 3 | ✓ ✓ | not published |
| 4 | ✓ ✓ | not published |
| 5 | ✓ ✓ | not published |
| 6 | ✓ ✓ | not published |
| 7 | ✓ ✓ | not published |
| 8 | ✓ ✓ | not published |
| 9 | ✓ ✓ | not published |
| 10 | ✓ ✓ | not published |
| 11 | ✓ ✓ | not published |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations |
| 9 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_insert_function (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_insert_function (✓ ✓) |
| 18 | — | semedit/semantic_insert_function (✓ ✓) |
| 19 | — | semedit/semantic_replace_body (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | ✓ ✓ |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_verify (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_insert_function (✓ ✓) |
| 32 | — | semedit/semantic_replace_body (✓ ✓) |
| 33 | — | semedit/semantic_replace_body (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | ✓ ✓ |
| 36 | — | semedit/semantic_verify (✓ ✓) |
| 37 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 117.16s | 179.03s | +52.8% | 131.83s | — | — |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | — | — |
| First Event → First Tool Call | 8.91s | 10.87s | — | 6.37s | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 2 | — | — |
| Internal Tool Cycles | 10 | 31 | +210.0% | 10 | — | — |
| Initial Load / Discovery Turns | 4 | 3 | -25.0% | 3 | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | — | — |
| Tool Invocations (per run) | 10 | 31 | +210.0% | 10 | — | — |
| Output Tokens | 13752 | 14465 | +5.2% | 9748 | — | — |
| Reasoning / Thinking Tokens | 6606 | 6354 | -3.8% | 5308 | — | — |
| Total Input Tokens | 1758719 | 3497480 | +98.9% | 786942 | — | — |
| Cached Input Tokens | 1617408 | 3294720 | +103.7% | 722944 | — | — |
| Uncached Input Tokens | 141311 | 202760 | +43.5% | 63998 | — | — |
| Cost | 1.0121 | 1.5908 | +57.2% | 0.5289 | — | — |
| Cached vs Uncached Token Ratio | 11.45:1 | 16.25:1 | +42.0% | 11.30:1 | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | — | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | PASS | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | — | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ? ✗ |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | ✓ ✓ |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | ✓ ✓ |
| 28 | — | semedit/semantic_replace_body (✓ ✓) |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_verify (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | not published |
| 2 | ✓ ✓ | not published |
| 3 | ✓ ✓ | not published |
| 4 | ✓ ✓ | not published |
| 5 | ✓ ✓ | not published |
| 6 | ✓ ✓ | not published |
| 7 | ✓ ✓ | not published |
| 8 | ✓ ✓ | not published |
| 9 | ✓ ✓ | not published |
| 10 | ✓ ✓ | not published |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 115.59s | 214.63s | +85.7% | 167.44s | 249.19s | +48.8% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 7.72s | 7.62s | — | 8.51s | 8.43s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 13 | 25 | +92.3% | 13 | 39 | +200.0% |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 13 | 25 | +92.3% | 13 | 39 | +200.0% |
| Output Tokens | 13206 | 19114 | +44.7% | 22620 | 25595 | +13.2% |
| Reasoning / Thinking Tokens | 6040 | 10621 | +75.8% | 10199 | 13192 | +29.3% |
| Total Input Tokens | 2476480 | 3275653 | +32.3% | 2976030 | 5663073 | +90.3% |
| Cached Input Tokens | 2320896 | 3081472 | +32.8% | 2759680 | 5408768 | +96.0% |
| Uncached Input Tokens | 155584 | 194181 | +24.8% | 216350 | 254305 | +17.5% |
| Cost | 1.2098 | 1.6275 | +34.5% | 1.6410 | 2.4728 | +50.7% |
| Cached vs Uncached Token Ratio | 14.92:1 | 15.87:1 | +6.4% | 12.76:1 | 21.27:1 | +66.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | PASS | — | FAIL | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ? ✗ | ✓ ✓ |
| 12 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 13 | ✓ ✓ | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 20 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ? ✗ |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Go verification over the entire workspace may format and modify tests or protected files, conflicting with the user's explicit prohibition on test edits and module changes. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that r |
| 20 | — | semedit/semantic_verify (✓ ✓) |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | semedit/semantic_verify (✓ ✓) |
| 28 | — | ✓ ✓ |
| 29 | — | ✓ ✓ |
| 30 | — | semedit/semantic_verify (✓ ✓) |
| 31 | — | ✓ ✓ |
| 32 | — | ✓ ✓ |
| 33 | — | semedit/semantic_verify (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | semedit/semantic_verify (✓ ✓) |
| 36 | — | semedit/semantic_replace_body (✓ ✓) |
| 37 | — | ✓ ✓ |
| 38 | — | ✓ ✓ |
| 39 | — | semedit/semantic_verify (✓ ✓) |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.02s | 119.83s | +149.5% | 110.62s | 189.80s | +71.6% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 5.54s | 8.20s | — | 13.41s | 8.90s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 5 | +400.0% |
| Internal Tool Cycles | 6 | 19 | +216.7% | 8 | 30 | +275.0% |
| Initial Load / Discovery Turns | 1 | 2 | +100.0% | 2 | 2 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 6 | 19 | +216.7% | 8 | 30 | +275.0% |
| Output Tokens | 1601 | 2482 | +55.0% | 4233 | 19756 | +366.7% |
| Reasoning / Thinking Tokens | 508 | 1045 | +105.7% | 2018 | 11025 | +446.3% |
| Total Input Tokens | 174020 | 658141 | +278.2% | 377052 | 3705799 | +882.8% |
| Cached Input Tokens | 157696 | 611328 | +287.7% | 347136 | 3486464 | +904.4% |
| Uncached Input Tokens | 16324 | 46813 | +186.8% | 29916 | 219335 | +633.2% |
| Cost | 0.1066 | 0.3140 | +194.5% | 0.2397 | 1.8047 | +652.9% |
| Cached vs Uncached Token Ratio | 9.66:1 | 13.06:1 | +35.2% | 11.60:1 | 15.90:1 | +37.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 25 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | semedit/semantic_replace_body (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | ✓ ✓ |
| 27 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 28 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 29 | — | semedit/semantic_replace_body (✓ ✓) |
| 30 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 105.97s | 135.06s | +27.5% | 269.38s | 281.80s | +4.6% |
| Process Start → First Event | 0.14s | 0.12s | — | 0.15s | 0.12s | — |
| First Event → First Tool Call | 5.53s | 9.39s | — | 10.07s | 9.70s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 10 | 17 | +70.0% | 14 | 40 | +185.7% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 10 | 17 | +70.0% | 14 | 40 | +185.7% |
| Output Tokens | 13742 | 10965 | -20.2% | 22213 | 26283 | +18.3% |
| Reasoning / Thinking Tokens | 7880 | 4006 | -49.2% | 10450 | 11444 | +9.5% |
| Total Input Tokens | 1445947 | 3213664 | +122.3% | 3160774 | 5241714 | +65.8% |
| Cached Input Tokens | 1324288 | 3033856 | +129.1% | 2977280 | 4967424 | +66.8% |
| Uncached Input Tokens | 121659 | 179808 | +47.8% | 183494 | 274290 | +49.5% |
| Cost | 0.9055 | 1.3951 | +54.1% | 1.6113 | 2.3992 | +48.9% |
| Cached vs Uncached Token Ratio | 10.89:1 | 16.87:1 | +55.0% | 16.23:1 | 18.11:1 | +11.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | FAIL | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | ✓ ✓ |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_function", "status": "error", "error": "invalid arguments: unexpected end of JSON input" } ], "final_diff": "" } |
| 7 | ✓ ✓ | semedit/report_feedback (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 14 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | ? ✗ |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 25 | — | semedit/semantic_replace_body (✓ ✓) |
| 26 | — | ✓ ✓ |
| 27 | — | semedit/semantic_verify (✓ ✓) |
| 28 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 29 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 30 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_verify (✓ ✓) |
| 33 | — | ✓ ✓ |
| 34 | — | semedit/semantic_insert_function (✓ ✓) |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | semedit/semantic_replace_body (✓ ✓) |
| 37 | — | semedit/semantic_replace_body (✓ ✓) |
| 38 | — | ✓ ✓ |
| 39 | — | semedit/semantic_verify (✓ ✓) |
| 40 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 197.85s | 131.14s | -33.7% | 153.63s | 272.82s | +77.6% |
| Process Start → First Event | 0.12s | 0.13s | — | 0.13s | 0.12s | — |
| First Event → First Tool Call | 11.04s | 5.25s | — | 9.60s | 10.29s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 19 | 22 | +15.8% | 13 | 37 | +184.6% |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | 3 | 4 | +33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 19 | 22 | +15.8% | 13 | 37 | +184.6% |
| Output Tokens | 30947 | 13214 | -57.3% | 21062 | 30405 | +44.4% |
| Reasoning / Thinking Tokens | 11984 | 6244 | -47.9% | 11232 | 15160 | +35.0% |
| Total Input Tokens | 3182547 | 2818630 | -11.4% | 2684911 | 5181938 | +93.0% |
| Cached Input Tokens | 2991360 | 2629376 | -12.1% | 2427648 | 4923392 | +102.8% |
| Uncached Input Tokens | 191187 | 189254 | -1.0% | 257263 | 258546 | +0.5% |
| Cost | 1.7624 | 1.3737 | -22.1% | 1.6537 | 2.4468 | +48.0% |
| Cached vs Uncached Token Ratio | 15.65:1 | 13.89:1 | -11.2% | 9.44:1 | 19.04:1 | +101.8% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | FAIL | — | PASS | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ? ✗ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | ✓ ✓ |
| 13 | ✓ ✓ | ✓ ✓ |
| 14 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | ✓ ✓ | ✓ ✓ |
| 16 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | ✓ ✓ | ✓ ✓ |
| 18 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | ✓ ✓ |
| 22 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 21 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification runs formatting that may write files, including forbidden test files; use a read-only or narrowly scoped verification instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains bl |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | ✓ ✓ |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | semedit/semantic_replace_body (✓ ✓) |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | ✓ ✓ |
| 32 | — | ✓ ✓ |
| 33 | — | semedit/semantic_insert_function (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | semedit/semantic_replace_body (✓ ✓) |
| 37 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 121.92s | 123.19s | +1.0% | 128.28s | 274.53s | +114.0% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.14s | — |
| First Event → First Tool Call | 8.46s | 7.91s | — | 10.22s | 9.66s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 5 | 5 | 0% |
| Internal Tool Cycles | 14 | 17 | +21.4% | 10 | 36 | +260.0% |
| Initial Load / Discovery Turns | 3 | 3 | 0% | 3 | 4 | +33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 14 | 17 | +21.4% | 10 | 36 | +260.0% |
| Output Tokens | 16378 | 13564 | -17.2% | 18819 | 31415 | +66.9% |
| Reasoning / Thinking Tokens | 7021 | 6955 | -0.9% | 10339 | 15479 | +49.7% |
| Total Input Tokens | 2096827 | 2935460 | +40.0% | 2289112 | 5048478 | +120.5% |
| Cached Input Tokens | 1967616 | 2727936 | +38.6% | 2118144 | 4793856 | +126.3% |
| Uncached Input Tokens | 129211 | 207524 | +60.6% | 170968 | 254622 | +48.9% |
| Cost | 1.1074 | 1.4573 | +31.6% | 1.3214 | 2.4212 | +83.2% |
| Cached vs Uncached Token Ratio | 15.23:1 | 13.15:1 | -13.7% | 12.39:1 | 18.83:1 | +52.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | FAIL | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | ✓ ✓ | ? ✗ |
| 13 | ✓ ✓ | ✓ ✓ |
| 14 | ✓ ✓ | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | ✓ ✓ |
| 17 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 23 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✗): validate access modifier: unsupported access modifier: "package-private" is not supported by the Go language backend (supported: infer, public, private) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | semedit/report_feedback (✓ ✓) |
| 11 | — | semedit/semantic_insert_function (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | ✓ ✓ |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | ✓ ✓ |
| 26 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 27 | — | semedit/semantic_replace_body (✓ ✓) |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | semedit/semantic_replace_body (✓ ✓) |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 103.63s | 290.60s | +180.4% | 57.02s | 269.11s | +371.9% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.12s | 0.12s | — |
| First Event → First Tool Call | 8.65s | 10.64s | — | 6.16s | 6.08s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 1 | 5 | +400.0% |
| Internal Tool Cycles | 11 | 30 | +172.7% | 8 | 47 | +487.5% |
| Initial Load / Discovery Turns | 1 | 5 | +400.0% | 1 | 2 | +100.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | 30 | +172.7% | 8 | 47 | +487.5% |
| Output Tokens | 12181 | 15159 | +24.4% | 1980 | 27001 | +1263.7% |
| Reasoning / Thinking Tokens | 4832 | 4856 | +0.5% | 703 | 16307 | +2219.6% |
| Total Input Tokens | 1873778 | 4741794 | +153.1% | 356382 | 3482764 | +877.3% |
| Cached Input Tokens | 1749760 | 4514816 | +158.0% | 332288 | 3225600 | +870.7% |
| Uncached Input Tokens | 124018 | 226978 | +83.0% | 24094 | 257164 | +967.3% |
| Cost | 0.9601 | 1.9463 | +102.7% | 0.1768 | 1.9907 | +1025.7% |
| Cached vs Uncached Token Ratio | 14.11:1 | 19.89:1 | +41.0% | 13.79:1 | 12.54:1 | -9.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | FAIL | FAIL | — | PASS | FAIL | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✗): unknown param "file"; supported keys are [bash-bin, bash_bin, ghc-bin, ghc-version, ghc_bin, ghc_version, haskell-standalone, hls-bin, hls-version, hls_bin, hls_version, import-maven, import_maven, include-tests, include-unexported, include_tests, include_unexported, java-bin, java-version, java_bin, java_version, jdtls-home, jdtls_home, kind, kinds, kotlin-bin, kotlin_bin, language, make-bin, make_bin, metals-bin, metals-home, metals_bin, metals_home, path, standalone_haskell, trust-workspace, trust_worksp |
| 5 | ? ✗ | semedit/report_feedback (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | ✓ ✓ |
| 16 | — | ✓ ✓ |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_verify (✓ ✓) |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_rename (✓ ✗): unknown param "new_name"; supported keys are [auto_organize_imports, file, import-maven, import_maven, java-bin, java_bin, jdtls-home, jdtls_home, language, symbol, to, trust-workspace, trust_workspace]: invalid operation parameters |
| 21 | — | semedit/report_feedback (✓ ✓) |
| 22 | — | semedit/semantic_rename (✓ ✓) |
| 23 | — | semedit/semantic_replace_body (✓ ✓) |
| 24 | — | semedit/semantic_replace_body (✓ ✓) |
| 25 | — | semedit/report_feedback (✓ ✓) |
| 26 | — | semedit/semantic_replace_body (✓ ✓) |
| 27 | — | semedit/semantic_verify (✓ ✓) |
| 28 | — | ✓ ✓ |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 25 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 9 | — | semedit/semantic_insert_function (✓ ✓) |
| 10 | — | semedit/semantic_replace_body (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Verification over the entire workspace is not read-only: it may format and write files, including tests that the user explicitly forbade editing; the requested scope is broader than authorized. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. R |
| 19 | — | ✓ ✓ |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | semedit/semantic_verify (✓ ✓) |
| 26 | — | semedit/semantic_verify (✓ ✓) |
| 27 | — | semedit/semantic_verify (✓ ✓) |
| 28 | — | semedit/semantic_verify (✓ ✓) |
| 29 | — | ✓ ✓ |
| 30 | — | ✓ ✓ |
| 31 | — | ✓ ✓ |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | semedit/semantic_verify (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | semedit/semantic_outline (✓ ✓) |
| 36 | — | semedit/semantic_verify (✓ ✓) |
| 37 | — | ✓ ✓ |
| 38 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 39 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 40 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 41 | — | semedit/semantic_insert_function (✓ ✓) |
| 42 | — | semedit/semantic_replace_body (✓ ✓) |
| 43 | — | semedit/semantic_replace_body (✓ ✓) |
| 44 | — | semedit/semantic_replace_body (✓ ✓) |
| 45 | — | semedit/semantic_verify (✓ ✓) |
| 46 | — | ✓ ✓ |
| 47 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 133.25s | 115.57s | -13.3% | 164.29s | 279.10s | +69.9% |
| Process Start → First Event | 0.16s | 0.12s | — | 0.15s | 0.14s | — |
| First Event → First Tool Call | 8.24s | 5.45s | — | 4.87s | 5.62s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 1 | -80.0% | 5 | 5 | 0% |
| Internal Tool Cycles | 12 | 17 | +41.7% | 17 | 47 | +176.5% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 7 | 4 | -42.9% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 12 | 17 | +41.7% | 17 | 47 | +176.5% |
| Output Tokens | 16626 | 2949 | -82.3% | 19980 | 23963 | +19.9% |
| Reasoning / Thinking Tokens | 8509 | 1513 | -82.2% | 9158 | 11582 | +26.5% |
| Total Input Tokens | 1437104 | 597371 | -58.4% | 2820512 | 4777618 | +69.4% |
| Cached Input Tokens | 1281792 | 552448 | -56.9% | 2604288 | 4522240 | +73.6% |
| Uncached Input Tokens | 155312 | 44923 | -71.1% | 216224 | 255378 | +18.1% |
| Cost | 1.0229 | 0.3062 | -70.1% | 1.5559 | 2.2133 | +42.3% |
| Cached vs Uncached Token Ratio | 8.25:1 | 12.30:1 | +49.0% | 12.04:1 | 17.71:1 | +47.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 10 | ✓ ✓ | ✓ ✓ |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_verify (✓ ✓) |
| 14 | — | semedit/semantic_verify (✓ ✓) |
| 15 | — | semedit/semantic_verify (✓ ✓) |
| 16 | — | semedit/semantic_verify (✓ ✓) |
| 17 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 9 | ✓ ✓ | semedit/report_feedback (✗ ?): This action was rejected due to unacceptable risk. Reason: The feedback call would transmit project paths, diagnostics, and implementation details to an unspecified external service; the user authorized source changes and semedit use, not this data disclosure. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirm |
| 10 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 14 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 15 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 16 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 17 | ✓ ✓ | semedit/semantic_organize_imports (✓ ✓) |
| 18 | — | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide verification runs formatting that can write files, potentially modifying forbidden tests or protected go.mod/go.sum; use a read-only or narrowly scoped verification instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report |
| 19 | — | semedit/semantic_verify (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
| 21 | — | semedit/semantic_verify (✓ ✓) |
| 22 | — | semedit/semantic_verify (✓ ✓) |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | semedit/semantic_verify (✓ ✓) |
| 25 | — | ✓ ✓ |
| 26 | — | ✓ ✓ |
| 27 | — | ✓ ✓ |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_verify (✓ ✓) |
| 30 | — | semedit/semantic_verify (✓ ✓) |
| 31 | — | semedit/semantic_verify (✓ ✓) |
| 32 | — | semedit/semantic_verify (✓ ✓) |
| 33 | — | semedit/semantic_verify (✓ ✓) |
| 34 | — | semedit/semantic_verify (✓ ✓) |
| 35 | — | ✓ ✓ |
| 36 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 37 | — | semedit/semantic_verify (✓ ✓) |
| 38 | — | ✓ ✓ |
| 39 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 40 | — | semedit/semantic_verify (✓ ✓) |
| 41 | — | ✓ ✓ |
| 42 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 43 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 44 | — | semedit/semantic_rename (✓ ✓) |
| 45 | — | semedit/semantic_replace_body (✓ ✓) |
| 46 | — | ✓ ✓ |
| 47 | — | ✓ ✓ |
prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | task-11-mixed-sink-api-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | prefer_discover_semedit |
| MCP server instructions | prescriptive |
| Semedit arm restriction | write |
| Fixture | testdata/bench/task_11_mixed_sink_api_migration.txtar |
Vanilla LLM Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.
Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 139.22s | 260.53s | +87.1% | 55.88s | 190.77s | +241.4% |
| Process Start → First Event | 0.12s | 0.12s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 7.47s | 6.67s | — | 5.67s | 13.45s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 5 | 5 | 0% | 1 | 2 | +100.0% |
| Internal Tool Cycles | 13 | 38 | +192.3% | 8 | 17 | +112.5% |
| Initial Load / Discovery Turns | 2 | 2 | 0% | 2 | 1 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 13 | 38 | +192.3% | 8 | 17 | +112.5% |
| Output Tokens | 16353 | 15512 | -5.1% | 1722 | 8241 | +378.6% |
| Reasoning / Thinking Tokens | 9437 | 6212 | -34.2% | 536 | 4460 | +732.1% |
| Total Input Tokens | 1873342 | 3888777 | +107.6% | 191664 | 1406447 | +633.8% |
| Cached Input Tokens | 1754112 | 3675648 | +109.5% | 173824 | 1325312 | +662.4% |
| Uncached Input Tokens | 119230 | 213129 | +78.8% | 17840 | 81135 | +354.8% |
| Cost | 1.0590 | 1.7233 | +62.7% | 0.1163 | 0.6929 | +495.9% |
| Cached vs Uncached Token Ratio | 14.71:1 | 17.25:1 | +17.2% | 9.74:1 | 16.33:1 | +67.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | FAIL | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | ✓ ✓ | ✓ ✓ |
| 10 | ✓ ✓ | ? ✗ |
| 11 | ✓ ✓ | semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification performs formatting that may write files, potentially including protected tests, despite the user's explicit no-test-edits constraint; a narrower read-only verification is safer. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without aski |
| 12 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_verify (✓ ✓) |
| 14 | — | ✓ ✓ |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_verify (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | ✓ ✓ |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_verify (✓ ✓) |
| 24 | — | ✓ ✓ |
| 25 | — | semedit/semantic_outline (✓ ✓) |
| 26 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 27 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 28 | — | ✓ ✓ |
| 29 | — | semedit/semantic_verify (✓ ✓) |
| 30 | — | ✓ ✓ |
| 31 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 32 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 33 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 34 | — | ✓ ✓ |
| 35 | — | semedit/semantic_replace_body (✓ ✓) |
| 36 | — | ✓ ✓ |
| 37 | — | semedit/semantic_verify (✓ ✓) |
| 38 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_insert_function (✓ ✓) |
| 6 | ? ✗ | semedit/semantic_replace_body (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_replace_body (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | ✓ ✓ |
| 11 | — | ✓ ✓ |
| 12 | — | ✓ ✓ |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_insert_function (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | ✓ ✓ |