Benchmark run task-12-strengthened-readwrite-20260929
This page contains exactly the publishable benchmark observations recorded in run task-12-strengthened-readwrite-20260929; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: task-12-request-identity-migration
Target: codex/gpt-6-luna/medium
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 87.12s | 103.39s | +18.7% | 58.40s | 155.44s | +166.2% |
| Process Start → First Event | 1.07s | 0.56s | — | 0.14s | 0.15s | — |
| First Event → First Tool Call | 12.62s | 10.77s | — | 4.96s | 9.25s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 18 | +157.1% | 8 | 23 | +187.5% |
| Initial Load / Discovery Turns | 3 | 5 | +66.7% | 3 | 3 | 0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 18 | +157.1% | 8 | 23 | +187.5% |
| Output Tokens | 2349 | 1699 | -27.7% | 2054 | 1958 | -4.7% |
| Reasoning / Thinking Tokens | 546 | 663 | +21.4% | 669 | 764 | +14.2% |
| Total Input Tokens | 205688 | 393618 | +91.4% | 198676 | 449873 | +126.4% |
| Cached Input Tokens | 186112 | 352256 | +89.3% | 180992 | 415232 | +129.4% |
| Uncached Input Tokens | 19576 | 41362 | +111.3% | 17684 | 34641 | +95.9% |
| Cost | 0.1317 | 0.2210 | +67.9% | 0.1235 | 0.2244 | +81.7% |
| Cached vs Uncached Token Ratio | 9.51:1 | 8.52:1 | -10.4% | 10.23:1 | 11.99:1 | +17.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ? ✗ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_lookup (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_rename (✓ ✓) |
| 14 | — | semedit/semantic_batch (✓ ✓) |
| 15 | — | semedit/semantic_find_references (✓ ✓) |
| 16 | — | semedit/semantic_lookup (✓ ✓) |
| 17 | — | semedit/semantic_lookup (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✗): resolve request.Request: symbol not found |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_insert_decl (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_insert_function (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_find_references (✓ ✓) |
| 22 | — | ✓ ✓ |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 64.81s | 134.02s | +106.8% | 63.81s | 129.19s | +102.4% |
| Process Start → First Event | 0.13s | 0.14s | — | 0.15s | 0.15s | — |
| First Event → First Tool Call | 4.92s | 5.86s | — | 4.54s | 5.61s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 9 | 24 | +166.7% | 8 | 22 | +175.0% |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 9 | 24 | +166.7% | 8 | 22 | +175.0% |
| Output Tokens | 2131 | 2177 | +2.2% | 2141 | 1919 | -10.4% |
| Reasoning / Thinking Tokens | 638 | 997 | +56.3% | 634 | 763 | +20.3% |
| Total Input Tokens | 233494 | 475294 | +103.6% | 202971 | 474331 | +133.7% |
| Cached Input Tokens | 212480 | 429824 | +102.3% | 184064 | 422912 | +129.8% |
| Uncached Input Tokens | 21014 | 45470 | +116.4% | 18907 | 51419 | +172.0% |
| Cost | 0.1403 | 0.2608 | +85.9% | 0.1280 | 0.2678 | +109.3% |
| Cached vs Uncached Token Ratio | 10.11:1 | 9.45:1 | -6.5% | 9.74:1 | 8.22:1 | -15.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✗): resolve ID in request/request.go: symbol not found |
| 9 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_find_references (✓ ✓) |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 15 | — | semedit/semantic_insert_decl (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_insert_function (✓ ✓) |
| 19 | — | semedit/semantic_find_references (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | file_change (✓ ✓) | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_find_references (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_rename (✓ ✓) |
| 11 | — | semedit/semantic_insert_decl (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_insert_function (✓ ✓) |
| 15 | — | semedit/semantic_find_references (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_lookup (✓ ✓) |
| 22 | — | semedit/semantic_lookup (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 71.89s | 80.79s | +12.4% | 60.14s | 104.49s | +73.8% |
| Process Start → First Event | 0.15s | 0.59s | — | 0.15s | 0.15s | — |
| First Event → First Tool Call | 4.45s | 7.95s | — | 7.06s | 7.61s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 13 | 14 | +7.7% | 7 | 17 | +142.9% |
| Initial Load / Discovery Turns | 5 | 3 | -40.0% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 13 | 14 | +7.7% | 7 | 17 | +142.9% |
| Output Tokens | 2550 | 1629 | -36.1% | 1811 | 1709 | -5.6% |
| Reasoning / Thinking Tokens | 628 | 600 | -4.5% | 393 | 620 | +57.8% |
| Total Input Tokens | 321606 | 377639 | +17.4% | 177975 | 439156 | +146.8% |
| Cached Input Tokens | 299520 | 328192 | +9.6% | 160768 | 388864 | +141.9% |
| Uncached Input Tokens | 22086 | 49447 | +123.9% | 17207 | 50292 | +192.3% |
| Cost | 0.1698 | 0.2335 | +37.5% | 0.1108 | 0.2521 | +127.6% |
| Cached vs Uncached Token Ratio | 13.56:1 | 6.64:1 | -51.1% | 9.34:1 | 7.73:1 | -17.2% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | ✓ ✓ |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_find_references (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 12 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 13 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 14 | — | semedit/semantic_find_references (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | ✓ ✓ |
| 10 | — | semedit/semantic_lookup (✓ ✓) |
| 11 | — | semedit/semantic_find_references (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_batch (✓ ✓) |
| 14 | — | semedit/semantic_find_references (✓ ✓) |
| 15 | — | ✓ ✓ |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 54.39s | 132.43s | +143.5% | 64.94s | 122.87s | +89.2% |
| Process Start → First Event | 0.13s | 0.13s | — | 0.13s | 0.53s | — |
| First Event → First Tool Call | 5.61s | 10.79s | — | 5.73s | 4.32s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 7 | 22 | +214.3% | 8 | 19 | +137.5% |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | 5 | 2 | -60.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 7 | 22 | +214.3% | 8 | 19 | +137.5% |
| Output Tokens | 1965 | 1889 | -3.9% | 1631 | 1657 | +1.6% |
| Reasoning / Thinking Tokens | 282 | 563 | +99.6% | 489 | 754 | +54.2% |
| Total Input Tokens | 180826 | 518540 | +186.8% | 195703 | 330519 | +68.9% |
| Cached Input Tokens | 161792 | 470528 | +190.8% | 176896 | 293120 | +65.7% |
| Uncached Input Tokens | 19034 | 48012 | +152.2% | 18807 | 37399 | +98.9% |
| Cost | 0.1161 | 0.2683 | +131.1% | 0.1177 | 0.1969 | +67.2% |
| Cached vs Uncached Token Ratio | 8.50:1 | 9.80:1 | +15.3% | 9.41:1 | 7.84:1 | -16.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_assertion_mode (✓ ✗): read assertion source .: read /Users/alessandro/.codex/worktrees/5ed4eaeb-36d5-4c10-aa76-b12d80abec0e/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-12-request-identity-migration_1790660929311259000: is a directory |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ? ✗ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_find_references (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_lookup (✓ ✓) |
| 13 | — | semedit/semantic_rename (✓ ✓) |
| 14 | — | semedit/semantic_insert_decl (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_insert_function (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_find_references (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_rename (✓ ✓) |
| 11 | — | semedit/semantic_insert_decl (✓ ✓) |
| 12 | — | semedit/semantic_replace_body (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_insert_function (✓ ✓) |
| 15 | — | semedit/semantic_find_references (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 59.64s | 131.65s | +120.8% | 50.42s | 91.96s | +82.4% |
| Process Start → First Event | 0.14s | 0.14s | — | 0.14s | 0.14s | — |
| First Event → First Tool Call | 6.56s | 8.11s | — | 5.61s | 6.38s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 26 | +225.0% | 6 | 14 | +133.3% |
| Initial Load / Discovery Turns | 3 | 6 | +100.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 26 | +225.0% | 6 | 14 | +133.3% |
| Output Tokens | 2037 | 2001 | -1.8% | 1602 | 1901 | +18.7% |
| Reasoning / Thinking Tokens | 373 | 772 | +107.0% | 330 | 931 | +182.1% |
| Total Input Tokens | 180548 | 503280 | +178.8% | 177090 | 450662 | +154.5% |
| Cached Input Tokens | 162816 | 445440 | +173.6% | 160768 | 408320 | +154.0% |
| Uncached Input Tokens | 17732 | 57840 | +226.2% | 16322 | 42342 | +159.4% |
| Cost | 0.1152 | 0.2906 | +152.4% | 0.1051 | 0.2433 | +131.4% |
| Cached vs Uncached Token Ratio | 9.18:1 | 7.70:1 | -16.1% | 9.85:1 | 9.64:1 | -2.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | ✓ ✓ |
| 6 | ✓ ✓ | ✓ ✓ |
| 7 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_lookup (✓ ✓) |
| 10 | — | semedit/semantic_lookup (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 14 | — | semedit/semantic_find_references (✓ ✓) |
| 15 | — | semedit/semantic_rename (✓ ✓) |
| 16 | — | semedit/semantic_insert_decl (✓ ✓) |
| 17 | — | semedit/semantic_replace_body (✓ ✓) |
| 18 | — | semedit/semantic_replace_body (✓ ✓) |
| 19 | — | semedit/semantic_insert_function (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | semedit/semantic_find_references (✓ ✓) |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | semedit/semantic_find_references (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_rename (✓ ✓) |
| 10 | — | semedit/semantic_insert_decl (✓ ✓) |
| 11 | — | semedit/semantic_batch (✓ ✓) |
| 12 | — | semedit/semantic_find_references (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | ✓ ✓ |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 51.17s | 114.88s | +124.5% | 55.86s | 176.23s | +215.5% |
| Process Start → First Event | 0.13s | 0.14s | — | 0.13s | 0.14s | — |
| First Event → First Tool Call | 4.49s | 5.25s | — | 4.81s | 11.99s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 22 | +175.0% | 6 | 25 | +316.7% |
| Initial Load / Discovery Turns | 4 | 1 | -75.0% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 22 | +175.0% | 6 | 25 | +316.7% |
| Output Tokens | 1979 | 1770 | -10.6% | 2203 | 2563 | +16.3% |
| Reasoning / Thinking Tokens | 227 | 707 | +211.5% | 311 | 1390 | +346.9% |
| Total Input Tokens | 220711 | 285091 | +29.2% | 155432 | 493828 | +217.7% |
| Cached Input Tokens | 203264 | 241664 | +18.9% | 138496 | 442112 | +219.2% |
| Uncached Input Tokens | 17447 | 43427 | +148.9% | 16936 | 51716 | +205.4% |
| Cost | 0.1220 | 0.1999 | +63.9% | 0.1084 | 0.2892 | +166.8% |
| Cached vs Uncached Token Ratio | 11.65:1 | 5.56:1 | -52.2% | 8.18:1 | 8.55:1 | +4.5% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_assertion_mode (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | semedit/semantic_outline (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_rename (✓ ✓) |
| 10 | — | semedit/semantic_insert_decl (✓ ✓) |
| 11 | — | semedit/semantic_replace_body (✓ ✓) |
| 12 | — | semedit/semantic_insert_function (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_find_references (✓ ✓) |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 18 | — | semedit/semantic_outline (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_assertion_mode (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_find_references (✓ ✓) |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_insert_decl (✓ ✓) |
| 14 | — | semedit/semantic_insert_function (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_replace_body (✓ ✓) |
| 17 | — | semedit/semantic_find_references (✓ ✓) |
| 18 | — | ✓ ✓ |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 25 | — | semedit/semantic_lookup (✓ ✗): resolve Request.ID: symbol not found |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 48.47s | 125.75s | +159.5% | 59.15s | 102.15s | +72.7% |
| Process Start → First Event | 0.14s | 0.13s | — | 0.14s | 0.15s | — |
| First Event → First Tool Call | 5.04s | 7.05s | — | 6.22s | 6.97s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 5 | 18 | +260.0% | 7 | 20 | +185.7% |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | 3 | 1 | -66.7% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 5 | 18 | +260.0% | 7 | 20 | +185.7% |
| Output Tokens | 1792 | 2205 | +23.0% | 1610 | 1499 | -6.9% |
| Reasoning / Thinking Tokens | 153 | 1095 | +615.7% | 393 | 515 | +31.0% |
| Total Input Tokens | 130080 | 282425 | +117.1% | 172616 | 473102 | +174.1% |
| Cached Input Tokens | 115200 | 237568 | +106.2% | 156672 | 424704 | +171.1% |
| Uncached Input Tokens | 14880 | 44857 | +201.5% | 15944 | 48398 | +203.5% |
| Cost | 0.0903 | 0.2128 | +135.6% | 0.1041 | 0.2523 | +142.5% |
| Cached vs Uncached Token Ratio | 7.74:1 | 5.30:1 | -31.6% | 9.83:1 | 8.78:1 | -10.7% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_find_references (✓ ✓) |
| 10 | — | ✓ ✓ |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_batch (✓ ✓) |
| 14 | — | semedit/semantic_find_references (✓ ✓) |
| 15 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 16 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_assertion_mode (✓ ✗): param "file" is required: invalid operation parameters |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 5 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_find_references (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | — | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_replace_body (✓ ✓) |
| 14 | — | semedit/semantic_insert_decl (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_insert_function (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_find_references (✓ ✓) |
| 19 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 20 | — | semedit/semantic_verify (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 51.46s | 78.89s | +53.3% | 54.05s | 102.84s | +90.3% |
| Process Start → First Event | 0.14s | 0.14s | — | 0.13s | 0.13s | — |
| First Event → First Tool Call | 4.76s | 7.49s | — | 4.71s | 8.03s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 11 | 15 | +36.4% | 7 | 18 | +157.1% |
| Initial Load / Discovery Turns | 3 | 2 | -33.3% | 3 | 2 | -33.3% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 11 | 15 | +36.4% | 7 | 18 | +157.1% |
| Output Tokens | 1926 | 1502 | -22.0% | 2004 | 1735 | -13.4% |
| Reasoning / Thinking Tokens | 90 | 700 | +677.8% | 721 | 797 | +10.5% |
| Total Input Tokens | 199670 | 275756 | +38.1% | 188654 | 367571 | +94.8% |
| Cached Input Tokens | 166656 | 234496 | +40.7% | 152576 | 325120 | +113.1% |
| Uncached Input Tokens | 33014 | 41260 | +25.0% | 36078 | 42451 | +17.7% |
| Cost | 0.1494 | 0.1893 | +26.7% | 0.1624 | 0.2191 | +34.9% |
| Cached vs Uncached Token Ratio | 5.05:1 | 5.68:1 | +12.6% | 4.23:1 | 7.66:1 | +81.1% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 5 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ? ✗ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | file_change (✓ ✓) | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | ✓ ✓ | semedit/semantic_rename (✓ ✓) |
| 12 | — | semedit/semantic_batch (✓ ✓) |
| 13 | — | semedit/semantic_lookup (✓ ✗): resolve Request.ID: symbol not found |
| 14 | — | semedit/semantic_lookup (✓ ✓) |
| 15 | — | semedit/semantic_find_references (✓ ✓) |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_find_references (✓ ✓) |
| 8 | — | ✓ ✓ |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 11 | — | semedit/semantic_lookup (✓ ✓) |
| 12 | — | semedit/semantic_rename (✓ ✓) |
| 13 | — | semedit/semantic_batch (✓ ✓) |
| 14 | — | semedit/semantic_lookup (✓ ✓) |
| 15 | — | semedit/semantic_lookup (✓ ✓) |
| 16 | — | semedit/semantic_lookup (✓ ✓) |
| 17 | — | ✓ ✓ |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 55.32s | 124.66s | +125.4% | 63.38s | 121.42s | +91.6% |
| Process Start → First Event | 0.13s | 0.14s | — | 0.15s | 0.14s | — |
| First Event → First Tool Call | 5.89s | 8.53s | — | 5.46s | 7.70s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 8 | 27 | +237.5% | 10 | 12 | +20.0% |
| Initial Load / Discovery Turns | 3 | 1 | -66.7% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 8 | 27 | +237.5% | 10 | 12 | +20.0% |
| Output Tokens | 2023 | 1483 | -26.7% | 2103 | 2246 | +6.8% |
| Reasoning / Thinking Tokens | 434 | 243 | -44.0% | 450 | 1338 | +197.3% |
| Total Input Tokens | 203684 | 445039 | +118.5% | 249615 | 297475 | +19.2% |
| Cached Input Tokens | 169728 | 398336 | +134.7% | 228608 | 256768 | +12.3% |
| Uncached Input Tokens | 33956 | 46703 | +37.5% | 21007 | 40707 | +93.8% |
| Cost | 0.1580 | 0.2379 | +50.5% | 0.1416 | 0.2108 | +48.9% |
| Cached vs Uncached Token Ratio | 5.00:1 | 8.53:1 | +70.6% | 10.88:1 | 6.31:1 | -42.0% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | semedit/semantic_assertion_mode (✓ ✗): param "file" is required: invalid operation parameters |
| 3 | ✓ ✓ | ? ✗ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 7 | ✓ ✓ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 9 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 10 | — | semedit/semantic_find_references (✓ ✓) |
| 11 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | ✓ ✓ |
| 14 | — | semedit/semantic_rename (✓ ✓) |
| 15 | — | semedit/semantic_find_references (✓ ✓) |
| 16 | — | semedit/semantic_lookup (✓ ✓) |
| 17 | — | semedit/semantic_lookup (✓ ✓) |
| 18 | — | semedit/semantic_lookup (✓ ✓) |
| 19 | — | semedit/semantic_lookup (✓ ✗): resolve ErrInvalidRequest in request/request.go: symbol not found |
| 20 | — | semedit/semantic_replace_body (✓ ✓) |
| 21 | — | semedit/semantic_replace_body (✓ ✓) |
| 22 | — | semedit/semantic_replace_body (✓ ✓) |
| 23 | — | semedit/semantic_insert_function (✓ ✓) |
| 24 | — | semedit/semantic_insert_decl (✓ ✓) |
| 25 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 26 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 27 | — | semedit/semantic_find_references (✓ ✗): find_references: go: symbol not found: "Request.ID" |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | file_change (✓ ✓) | semedit/semantic_lookup (✓ ✓) |
| 6 | file_change (✓ ✓) | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | file_change (✓ ✓) | semedit/semantic_rename (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 11 | — | ✓ ✓ |
| 12 | — | semedit/semantic_find_references (✓ ✓) |
default prompt · prescriptive MCP instructions · readwrite semedit restriction
| Setting | Value |
|---|---|
| Test case | task-12-request-identity-migration |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | default |
| MCP server instructions | prescriptive |
| Semedit arm restriction | readwrite |
| Fixture | testdata/bench/task_12_request_identity_migration.txtar |
Vanilla LLM Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.
Semedit MCP Prompt:
You are forbidden to modify protected files: “go.mod”, “go.sum”.
Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not inspect Go implementation files (non-test .go) with shell commands such as cat, sed, or rg. Use semantic code inspection tools for those files. Reading project documentation (docs/.md), build manifests (go.mod, Makefile), and visible test files in the task workspace is permitted. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Initial Workspace State (Before Edit)
package request
import (
"errors"
"strings"
)
type Request struct {
ID string
Kind string
Payload []byte
}
func (r Request) Validate() error {
if r.ID == "" {
return errors.New("invalid request")
}
return nil
}
func Normalize(kind string) string {
kind = strings.TrimSpace(kind)
kind = strings.ToLower(kind)
return kind
}
func FindByID(requests []Request, id string) (Request, bool) {
for _, r := range requests {
if r.ID == id {
return r, true
}
}
return Request{}, false
}| Metric | Vanilla (Small) | MCP (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 58.65s | 90.86s | +54.9% | 47.78s | 151.40s | +216.9% |
| Process Start → First Event | 0.14s | 0.15s | — | 0.13s | 0.17s | — |
| First Event → First Tool Call | 5.71s | 10.75s | — | 4.57s | 10.02s | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 1 | 1 | 0% | 1 | 1 | 0% |
| Internal Tool Cycles | 10 | 13 | +30.0% | 8 | 24 | +200.0% |
| Initial Load / Discovery Turns | 4 | 2 | -50.0% | 4 | 2 | -50.0% |
| MCP Discovery / Schema Turns | 0 | 0 | — | 0 | 0 | — |
| Tool Invocations (per run) | 10 | 13 | +30.0% | 8 | 24 | +200.0% |
| Output Tokens | 2093 | 1953 | -6.7% | 1670 | 2288 | +37.0% |
| Reasoning / Thinking Tokens | 435 | 1051 | +141.6% | 404 | 1141 | +182.4% |
| Total Input Tokens | 243889 | 381750 | +56.5% | 198121 | 558846 | +182.1% |
| Cached Input Tokens | 223488 | 331008 | +48.1% | 179968 | 506880 | +181.7% |
| Uncached Input Tokens | 20401 | 50742 | +148.7% | 18153 | 51966 | +186.3% |
| Cost | 0.1385 | 0.2472 | +78.5% | 0.1163 | 0.2995 | +157.5% |
| Cached vs Uncached Token Ratio | 10.95:1 | 6.52:1 | -40.5% | 9.91:1 | 9.75:1 | -1.6% |
| Oracle L1: Mutation Policy | PASS | PASS | — | PASS | PASS | — |
| Oracle L2: AST Invariants | PASS | PASS | — | PASS | PASS | — |
| Oracle L3: Clean Build | PASS | PASS | — | PASS | PASS | — |
| Oracle L4: Verification Test | PASS | PASS | — | PASS | PASS | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | N/A | PASS | — |
Baseline vs Semedit Agentic in Small Context
- Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ? ✗ |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 5 | file_change (✓ ✓) | semedit/semantic_inspect_symbol (✓ ✓) |
| 6 | file_change (✓ ✓) | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ? ✗ | ✓ ✓ |
| 8 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 9 | ✓ ✓ | semedit/semantic_find_references (✓ ✓) |
| 10 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 11 | — | semedit/semantic_find_references (✓ ✓) |
| 12 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 13 | — | ✓ ✓ |
Baseline vs Semedit Agentic in Large Context
- Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
- Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ? ✗ | semedit/semantic_lookup (✓ ✓) |
| 4 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 5 | ✓ ✓ | semedit/semantic_lookup (✓ ✓) |
| 6 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 7 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 8 | ✓ ✓ | semedit/semantic_inspect_symbol (✓ ✓) |
| 9 | — | semedit/semantic_lookup (✓ ✓) |
| 10 | — | semedit/semantic_find_references (✓ ✓) |
| 11 | — | semedit/semantic_rename (✓ ✓) |
| 12 | — | ✓ ✓ |
| 13 | — | semedit/semantic_insert_decl (✓ ✓) |
| 14 | — | semedit/semantic_replace_body (✓ ✓) |
| 15 | — | semedit/semantic_replace_body (✓ ✓) |
| 16 | — | semedit/semantic_insert_function (✓ ✓) |
| 17 | — | semedit/semantic_find_references (✓ ✓) |
| 18 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 19 | — | semedit/semantic_lookup (✓ ✗): resolve Request.ID in request/request.go: symbol not found |
| 20 | — | ✓ ✓ |
| 21 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 22 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 23 | — | semedit/semantic_inspect_symbol (✓ ✓) |
| 24 | — | semedit/semantic_inspect_symbol (✓ ✓) |