Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run task-12-strengthened-write-20260929

Benchmark run task-12-strengthened-write-20260929

This page contains exactly the publishable benchmark observations recorded in run task-12-strengthened-write-20260929; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-12-request-identity-migration

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency52.34s97.49s+86.3%49.18s125.80s+155.8%
Process Start → First Event0.67s0.67s—0.15s0.15s—
First Event → First Tool Call6.41s7.44s—4.66s6.89s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles713+85.7%717+142.9%
Initial Load / Discovery Turns440%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)713+85.7%717+142.9%
Output Tokens15361857+20.9%17222113+22.7%
Reasoning / Thinking Tokens191815+326.7%366855+133.6%
Total Input Tokens173963341798+96.5%174710447422+156.1%
Cached Input Tokens154624297472+92.4%158720409600+158.1%
Uncached Input Tokens1933944326+129.2%1599037822+136.5%
Cost0.10860.2186+101.3%0.10580.2341+121.3%
Cached vs Uncached Token Ratio8.00:16.71:1-16.1%9.93:110.83:1+9.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -120 && \
rg -n 'type Request|func .*Validate|func .*Normalize|\\.ID\\b|ErrInvalidRequest' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat go.mod && \
rg --files -g 'Makefile' -g 'README*' -g '*.md'"
✓ ✓
/bin/zsh -lc "git status --short && \
rg -n 'type Request struct|func .*Normalize|ErrInvalidRequest|Request\\.ID|\\.ID' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc "cat docs/identity-model.md && \
cat AGENTS.override.md && \
rg -n '\\bRequest\\{' --glob '*.go' . && \
rg -n 'Request\\.[A-Z][A-Za-z0-9_]*|\\.Request\\.' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat request/request_test.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' -g 'README.md' ."
4✓ ✓
/bin/zsh -lc 'cat request/request_test.go; cat gateway/handler.go queue/queue.go storage/requests.go scheduler/jobs.go notifications/notifier.go reporting/report.go correlation/identities.go'
✓ ✓
/bin/zsh -lc 'cat gateway/handler.go storage/requests.go queue/queue.go scheduler/jobs.go notifications/notifier.go'
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
files = {
'request/request.go': [('ID      string','RequestID string'), ('if r.ID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")\\n\\t}', 'if r.RequestID == \"\" {\\n\\t\\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\\n\\t}'), ('\"errors\"','\"errors\"\\n\\t\"fmt\"'), ('type Request struct {','var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\ntype Request struct {'), ('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}'), ('r.ID == id','r.RequestID == id')],
'gateway/handler.go':[('r.ID','r.RequestID')],
'notifications/notifier.go':[('r.ID','r.RequestID'),('r.Request.ID','r.Request.RequestID')],
'scheduler/jobs.go':[('left.Request.ID','left.Request.RequestID'),('right.Request.ID','right.Request.RequestID'),('job.Request.ID','job.Request.RequestID'),('r.ID','r.RequestID')],
'queue/queue.go':[('r.ID','r.RequestID')],
'correlation/identities.go':[('r.ID, resp.ID','r.RequestID, resp.ID')],
'reporting/report.go':[('r.Request.ID','r.Request.RequestID')],
'storage/requests.go':[('record.ID == r.ID','record.ID == r.RequestID')],
'request/request_test.go':[('Request{\"req-7\", \"audit\", nil}', 'Request{RequestID: \"req-7\", Kind: \"audit\"}'), ('[]Request{{\"other\", \"audit\", nil}, want}', '[]Request{{RequestID: \"other\", Kind: \"audit\"}, want}')],
'gateway/handler_test.go':[('request.Request{\"req-1\", \"audit\", nil}', 'request.Request{RequestID: \"req-1\", Kind: \"audit\"}')],
'storage/requests_test.go':[('request.Request{\"req-2\", \"\", nil}', 'request.Request{RequestID: \"req-2\"}')],
}
for name, replacements in files.items():
 p=Path(name); s=p.read_text()
 for old,new in replacements:
  if old not in s: raise SystemExit(f'{name}: missing {old}')
  s=s.replace(old,new)
 p.write_text(s)
PY
gofmt -w request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go scheduler/jobs.go queue/queue.go correlation/identities.go reporting/report.go storage/requests.go storage/requests_test.go
"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Normalize"
}
6✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|Request\\{\"' --glob '*.go' . && \
git diff -- go.mod go.sum && \
git diff --stat && \
go test ./... && \
go build ./..."
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go scheduler/jobs.go queue/queue.go correlation/identities.go reporting/report.go storage/requests.go storage/requests_test.go && \
git diff -- docs/SUBOPTIMAL_TOOLS.md tools/benchmark-harness/oracle.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
8—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "(Request).Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
9—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "fmt"
  ],
  "file": "request/request.go"
}
10—✓ ✓
/bin/zsh -lc "rg -n '\\b(Request|request\\.Request)\\b|\\.ID\\b|RequestID|normalizeKind|ErrInvalidRequest' --glob '*.go' ."
11—✓ ✓
/bin/zsh -lc 'git diff -- request go.mod go.sum'
12—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.RequestID"
}
13—✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat go.mod && \
if test -f go.sum; then wc -l go.sum; fi'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
rg -n '\\bRequest\\b|\\.ID\\b|func Normalize|Validate|ErrInvalid' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
3? ✗
/bin/zsh -lc "cat request/request.go request/request_test.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
rg --files -g 'Makefile' -g 'README*' -g '*.sh'"
✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n '\\b(ID|RequestID|Normalize|Validate)\\b|\\.ID\\b|Request\\{' --glob '*.go'"
4file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request"
}
5✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|Request\\s*\\{' --glob '*.go'"
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
6file_change (✓ ✓)semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w request/request.go request/request_test.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
go build ./... && \
go test ./...'
semedit/semantic_insert_declaration (✓ ✓)
{
  "file": "request/request.go",
  "placement": "public_start",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")",
  "visibility": "public"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "placement": "private_end",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
11—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Normalize"
}
12—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.RequestID"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request"
}
14—✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff -- go.mod go.sum && \
git diff -- request/request.go'
15—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "ErrInvalidRequest"
}
16—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "normalizeKind"
}
17—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.Validate"
}

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency44.54s111.37s+150.1%70.44s114.91s+63.1%
Process Start → First Event0.14s0.13s—0.14s0.14s—
First Event → First Tool Call5.82s4.68s—6.00s4.61s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles716+128.6%917+88.9%
Initial Load / Discovery Turns31-66.7%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)716+128.6%917+88.9%
Output Tokens15371831+19.1%21193210+51.5%
Reasoning / Thinking Tokens349617+76.8%4711703+261.6%
Total Input Tokens175096454017+159.3%227720584226+156.6%
Cached Input Tokens158720416768+162.6%204288533504+161.2%
Uncached Input Tokens1637637249+127.5%2343250722+116.5%
Cost0.10420.2279+118.7%0.14200.3216+126.4%
Cached vs Uncached Token Ratio9.69:111.19:1+15.4%8.72:110.52:1+20.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg -n \"type Request struct|func \\(.*Request.*\\) Validate|func Normalize|\\.ID\\b|Request\\{.*ID|ErrInvalidRequest\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md' -g 'go.mod' -g 'go.sum' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -160"
2✓ ✓
/bin/zsh -lc "rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' && \
cat request/request.go && \
cat go.mod && \
rg -n \"\\bRequest\\b|request\\.\" --glob '*.go' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Normalize"
}
3✓ ✓
/bin/zsh -lc 'cat request/request_test.go && \
cat gateway/handler.go && \
cat docs/identity-model.md && \
'
✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; git status --short'
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
files = {
'request/request.go': [('ID      string','RequestID string'), ('r.ID','r.RequestID')],
'gateway/handler.go': [('r.ID','r.RequestID')],
'notifications/notifier.go': [('r.Request.ID','r.Request.RequestID'), ('r.ID','r.RequestID')],
'queue/queue.go': [('r.ID','r.RequestID')],
'storage/requests.go': [('r.ID','r.RequestID')],
'scheduler/jobs.go': [('left.Request.ID','left.Request.RequestID'), ('right.Request.ID','right.Request.RequestID'), ('job.Request.ID','job.Request.RequestID'), ('r.ID','r.RequestID')],
'reporting/report.go': [('r.Request.ID','r.Request.RequestID')],
}
for name, replacements in files.items():
    p=Path(name)
    data=p.read_text()
    for old,new in replacements:
        data=data.replace(old,new)
    p.write_text(data)

p=Path('request/request.go')
s=p.read_text()
s=s.replace('func (r Request) Validate() error {\\n\\tif r.RequestID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")\\n', 'var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\nfunc (r Request) Validate() error {\\n\\tif r.RequestID == \"\" {\\n\\t\\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\\n')
s=s.replace('\"errors\"\\n\\t\"strings\"', '\"errors\"\\n\\t\"fmt\"\\n\\t\"strings\"')
s=s.replace('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}')
p.write_text(s)
PY
rg -n \"\\.ID\\b|Request\\{.*ID\" --glob '*.go' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "language": "go",
  "symbol": "Request.Validate"
}
5✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat gateway/handler.go && \
cat notifications/notifier.go && \
cat queue/queue.go && \
cat storage/requests.go && \
cat scheduler/jobs.go && \
cat reporting/report.go && \
git status --short'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Request.ID"
}
6? ✗
/bin/zsh -lc 'gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go && \
go test ./...'
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
7✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p=Path('correlation/identities.go')
s=p.read_text().replace('return []string{r.ID, resp.ID, review.ID}', 'return []string{r.RequestID, resp.ID, review.ID}')
p.write_text(s)
PY
gofmt -w correlation/identities.go
go build ./... && \
go test ./..."
semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"%w: request ID is empty\", ErrInvalidRequest)\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
9—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "fmt"
  ],
  "file": "request/request.go"
}
10—semedit/semantic_insert_function (✓ ✗): unknown param "declaration"; supported keys are [access, access_modifier, auto-organize-imports, auto_organize_imports, file, placement, source, target, target_symbol]: invalid operation parameters
{
  "access_modifier": "private",
  "declaration": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
  "file": "request/request.go"
}
11—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
13—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.RequestID"
}
14—✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n 'Request\\s*\\{|\\.RequestID|\\.ID' --glob '*.go' ."
15—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.Validate"
}
16—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "normalizeKind"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' | head -80 && \
rg -n '\\bRequest\\b|\\.ID\\b|func .*Normalize|Validate\\(' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'tools/benchmark-harness/README.md' -g 'docs/adr/README.md' -g 'docs/research/README.md' | head -40"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat request/request_test.go && \
rg -n '\\.ID|Request\\{' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files | head -80 && \
rg -n 'type Request struct|func \\(.*Request.*Validate|func Normalize|Request\\{' --glob '*.go' | head -100"
3✓ ✓
/bin/zsh -lc 'rg --files | sort && \
cat go.mod'
✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat docs/identity-model.md && \
cat go.mod && \
rg -n '\\.ID|Request\\{|RequestRecord|Normalize\\(' --glob '*.go'"
4file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Request"
}
5file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Request.ID"
}
6file_change (✓ ✓)semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.Validate"
}
7✓ ✓
/bin/zsh -lc "rg -n '\\.ID|Request\\{' --glob '*.go' . && \
rg -n 'RequestID|Request\\.ID|r\\.ID' --glob '*.go' . && \
gofmt -w request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go queue/queue.go storage/requests.go storage/requests_test.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
go test ./... && \
go build ./..."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Normalize"
}
8✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- go.mod go.sum && \
git diff --check && \
git diff --stat && \
rg -n 'Request\\.ID|request\\.Request\\{["'^'"\\n]*,|\\.ID' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID"
}
9✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git check-ignore -v request/request.go || \
true && \
git diff -- request/request.go request/request_test.go gateway/handler.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
10—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "access_modifier": "public",
        "file": "request/request.go",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "auto_organize_imports": true,
        "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"request ID is empty: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
11—✓ ✓
/bin/zsh -lc "rg --files | rg 'SUBOPTIMAL_TOOLS\\.md"'$|'"\\.scratch' | head -30"
12—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "fmt"
  ],
  "file": "request/request.go"
}
13—file_change (✓ ✓)
14—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --check && \
git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go reporting/report.go scheduler/jobs.go correlation/identities.go storage/requests.go && \
git diff -- go.mod go.sum && \
cat request/request.go'
15—✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git diff --stat && \
git check-ignore -v request/request.go || \
true && \
git status --short --untracked-files=no'
16—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.RequestID"
}
17—✓ ✓
/bin/zsh -lc "git diff -- go.mod go.sum; rg -n 'Request\\.ID|Request\\.RequestID|resp\\.ID|review\\.ID' --glob '*.go'"

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency50.30s92.38s+83.6%55.18s78.58s+42.4%
Process Start → First Event0.15s0.15s—0.14s0.15s—
First Event → First Tool Call4.56s4.53s—5.23s4.53s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles119-18.2%811+37.5%
Initial Load / Discovery Turns52-60.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)119-18.2%811+37.5%
Output Tokens17922795+56.0%20381429-29.9%
Reasoning / Thinking Tokens2781824+556.1%386626+62.2%
Total Input Tokens219103390036+78.0%199038347022+74.3%
Cached Input Tokens198144338176+70.7%180992322304+78.1%
Uncached Input Tokens2095951860+147.4%1804624718+37.0%
Cost0.12780.2719+112.8%0.12070.1681+39.3%
Cached vs Uncached Token Ratio9.45:16.52:1-31.0%10.03:113.04:1+30.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*request*' | head -100"
2? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'README*' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.yml' -g '*.yaml'"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'docs/**' | head -80 && \
sed -n '1,240p' request/request.go && \
rg -n '\\bRequest\\b|\\.ID\\b|Request\\.ID' --glob '*.go'"
3✓ ✓
/bin/zsh -lc 'cat go.mod && \
cat request/request.go && \
cat request/request_test.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
4✓ ✓
/bin/zsh -lc "rg -n '\\bRequest\\b|\\.ID\\b|ErrInvalid|Normalize|Validate' --glob '*.go' --glob '"'!**/*_test.go'"'"
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
5✓ ✓
/bin/zsh -lc 'cat gateway/handler.go notifications/notifier.go queue/queue.go correlation/identities.go storage/requests.go reporting/report.go scheduler/jobs.go && \
cat response/response.go review/review.go'
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "access_modifier": "public",
        "file": "request/request.go",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "auto_organize_imports": true,
        "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"validate request: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "(*Request).Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
6file_change (✓ ✓)semedit/semantic_organize_imports (✓ ✓)
{
  "file": "request/request.go"
}
7file_change (✓ ✓)✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff --'
8✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|Request\\{["'^'"\\n]*ID|Request\\{ID:' --glob '*.go'; rg -n 'ErrInvalidRequest|Validate|SameRequest|FindByID' --glob '*_test.go'"
semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.RequestID"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go correlation/identities.go storage/requests.go reporting/report.go scheduler/jobs.go && \
go test ./...'
✓ ✓
/bin/zsh -lc "sed -n '1,100p' request/request.go && \
git status --short -- request/request.go gateway/handler.go notifications/notifier.go go.mod go.sum && \
git diff -- request/request.go gateway/handler.go notifications/notifier.go"
10file_change (✓ ✓)—
11✓ ✓
/bin/zsh -lc 'gofmt -w request/request_test.go && \
go build ./... && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum'
—
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*request*' | head -200 && \
rg -n '\\btype Request\\b|func \\(["'^)]*Request'"\\) Validate|func Normalize|\\.ID\\b' --glob '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat request/request_test.go && \
cat go.mod && \
rg --files'
✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat go.mod && \
rg -n '\\.ID|ID:' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat docs/identity-model.md && \
for f in gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go storage/requests.go correlation/identities.go reporting/report.go; do printf '\\n--- %s ---\\n' \""'$f"; cat "$f"; done'
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request"
}
4file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
5? ✗
/bin/zsh -lc "gofmt -w request/request.go request/request_test.go gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go storage/requests.go correlation/identities.go reporting/report.go && \
rg -n '\\bRequest\\s*\\{|\\.ID\\b|\\bID\\s+string|Normalize\\(' --glob '*.go' && \
go build ./... && \
go test ./..."
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
6file_change (✓ ✓)semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "group": "append",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w request/request_test.go && \
go build ./... && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "(*Request).Validate"
}
8✓ ✓
/bin/zsh -lc 'git diff -- request/request.go request/request_test.go gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go storage/requests.go correlation/identities.go reporting/report.go && \
git diff -- go.mod go.sum && \
git status --short -- request/request.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
9—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
10—✓ ✓
/bin/zsh -lc "git diff -- request/request.go && \
git status --short && \
rg -n 'RequestID|\\.ID' --glob '*.go' ."
11—✓ ✓
/bin/zsh -lc 'cat request/request.go; cat request/request_test.go; git diff -- go.mod go.sum; git rev-parse --show-toplevel; git status --short -- request/request.go request/request_test.go go.mod go.sum'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency42.74s85.19s+99.3%38.77s120.90s+211.9%
Process Start → First Event0.13s0.15s—0.14s0.15s—
First Event → First Tool Call5.63s6.45s—4.42s6.88s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles513+160.0%517+240.0%
Initial Load / Discovery Turns30-100.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)513+160.0%517+240.0%
Output Tokens1620759-53.1%13532369+75.1%
Reasoning / Thinking Tokens960-100.0%332970+192.2%
Total Input Tokens129732459970+254.6%127599406282+218.4%
Cached Input Tokens109056413696+279.3%113152359680+217.9%
Uncached Input Tokens2067646274+123.8%1444746602+222.6%
Cost0.10040.2286+127.7%0.08550.2482+190.4%
Cached vs Uncached Token Ratio5.27:18.94:1+69.5%7.83:17.72:1-1.5%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*request*' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100 && \
rg -n 'type Request|func \\(.*Request.*\\) Validate|func Normalize|\\.ID\\b|ErrInvalidRequest|Normalize\\(' --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request"
}
2✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat request/request.go; cat gateway/handler.go; cat notifications/notifier.go; cat queue/queue.go; cat storage/requests.go; cat scheduler/jobs.go; cat correlation/identities.go; cat reporting/report.go; cat go.mod; cat request/request_test.go'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": true,
  "path": "request"
}
3✓ ✓
/bin/zsh -lc "rg -n '\\bRequest\\s*\\{|request\\.Request|\\.ID\\b|FindByID|Validate\\(' --glob '*.go' ."
semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
files = {
'request/request.go': '''package request

import (
\\t\"errors\"
\\t\"fmt\"
\\t\"strings\"
)

var ErrInvalidRequest = errors.New(\"invalid request\")

type Request struct {
\\tRequestID string
\\tKind      string
\\tPayload   []byte
}

func (r Request) Validate() error {
\\tif r.RequestID == \"\" {
\\t\\treturn fmt.Errorf(\"request ID is empty: %w\", ErrInvalidRequest)
\\t}
\\treturn nil
}

func Normalize(kind string) string {
\\treturn normalizeKind(kind)
}

func normalizeKind(kind string) string {
\\tkind = strings.TrimSpace(kind)
\\tkind = strings.ToLower(kind)
\\treturn kind
}

func FindByID(requests []Request, id string) (Request, bool) {
\\tfor _, r := range requests {
\\t\\tif r.RequestID == id {
\\t\\t\\treturn r, true
\\t\\t}
\\t}
\\treturn Request{}, false
}
''',
'gateway/handler.go': None,
'notifications/notifier.go': None,
'queue/queue.go': None,
'storage/requests.go': None,
'scheduler/jobs.go': None,
'correlation/identities.go': None,
'reporting/report.go': None,
'gateway/handler_test.go': None,
'request/request_test.go': None,
'storage/requests_test.go': None,
}
for name in files:
    if name "'!= '"'request/request.go':
        p=Path(name); old=p.read_text(); new=old
        if name == 'gateway/handler.go': new=new.replace('r.ID, err)', 'r.RequestID, err)')
        elif name == 'notifications/notifier.go': new=new.replace('r.ID', 'r.RequestID').replace('r.Request.ID','r.Request.RequestID')
        elif name == 'queue/queue.go': new=new.replace('r.ID','r.RequestID')
        elif name == 'storage/requests.go': new=new.replace('record.ID == r.ID','record.ID == r.RequestID')
        elif name == 'scheduler/jobs.go': new=new.replace('left.Request.ID','left.Request.RequestID').replace('right.Request.ID','right.Request.RequestID').replace('job.Request.ID','job.Request.RequestID').replace('r.ID','r.RequestID')
        elif name == 'correlation/identities.go': new=new.replace('r.ID, resp.ID','r.RequestID, resp.ID')
        elif name == 'reporting/report.go': new=new.replace('r.Request.ID','r.Request.RequestID')
        elif name == 'gateway/handler_test.go': new=new.replace('request.Request{\"req-1\", \"audit\", nil}', 'request.Request{RequestID: \"req-1\", Kind: \"audit\"}')
        elif name == 'request/request_test.go': new=new.replace('Request{\"req-7\", \"audit\", nil}', 'Request{RequestID: \"req-7\", Kind: \"audit\"}').replace('Request{{\"other\", \"audit\", nil}, want}', 'Request{{RequestID: \"other\", Kind: \"audit\"}, want}')
        elif name == 'storage/requests_test.go': new=new.replace('request.Request{\"req-2\", \"\", nil}', 'request.Request{RequestID: \"req-2\"}')
        files[name]=new
for name, content in files.items():
    p=Path(name)
    if content is None: continue
    fd,tmp=tempfile.mkstemp(dir=p.parent, prefix='.'+p.name+'.')
    try:
        with os.fdopen(fd,'w') as f:
            f.write(content); f.flush(); os.fsync(f.fileno())
        os.replace(tmp,p)
    finally:
        if os.path.exists(tmp): os.unlink(tmp)
PY
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go gateway/handler_test.go request/request_test.go storage/requests_test.go"
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Validate"
}
5✓ ✓
/bin/zsh -lc "rg -n 'r\\.ID|Request\\.ID|Request\\{\"|Request\\{\\{' --glob '*.go' . && \
go build ./... && \
go test ./..."
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
7—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
8—semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "group": "append",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"request ID: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
11—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
12—semedit/semantic_outline (✓ ✓)
{
  "include_tests": true,
  "path": "."
}
13—semedit/semantic_outline (✓ ✓)
{
  "path": "request/request.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' | head -200 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -160 && \
git status --short"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n '\\.ID|\\bID\\b|Normalize|Validate' --glob '*.go' && \
rg --files -g 'README*' -g 'Makefile' -g 'go.mod' -g 'AGENTS*' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README.md' | head -80 && \
cat go.mod && \
rg -n 'func \\(.*Request.*\\) Validate|func Normalize|type Request|\\.ID\\b|Request\\s*\\{' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc 'cat gateway/handler.go; cat request/request_test.go; cat queue/queue.go notifications/notifier.go correlation/identities.go scheduler/jobs.go reporting/report.go storage/requests.go; cat Makefile 2>/dev/null; cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
root=Path('.')
changes={
'request/request.go': [('ID      string','RequestID string'),('if r.ID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")','if r.RequestID == \"\" {\\n\\t\\treturn ErrInvalidRequest'),('func (r Request) Validate() error {','var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\nfunc (r Request) Validate() error {'),('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}','func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}'),('r.ID == id','r.RequestID == id')],
'gateway/handler.go': [('r.ID, err','r.RequestID, err')],
'queue/queue.go': [('r.ID','r.RequestID')],
'notifications/notifier.go': [('r.ID','r.RequestID'),('r.Request.ID','r.Request.RequestID')],
'correlation/identities.go': [('r.ID, resp.ID','r.RequestID, resp.ID')],
'scheduler/jobs.go': [('left.Request.ID','left.Request.RequestID'),('right.Request.ID','right.Request.RequestID'),('job.Request.ID','job.Request.RequestID'),('r.ID','r.RequestID')],
'reporting/report.go': [('r.Request.ID','r.Request.RequestID')],
'storage/requests.go': [('r.ID','r.RequestID')],
}
for name,repls in changes.items():
 p=root/name
 text=p.read_text()
 for old,new in repls:
  if old not in text: raise SystemExit(f'missing {old"'!r} in {name}'"')
  text=text.replace(old,new)
 fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=p.name+'.')
 try:
  with os.fdopen(fd,'w') as f:
   f.write(text); f.flush(); os.fsync(f.fileno())
  os.replace(tmp,p)
 finally:
  if os.path.exists(tmp): os.unlink(tmp)
PY
"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Request.Validate"
}
5✓ ✓
/bin/zsh -lc "rg -n '\\.ID|\\bID\\s+string|ErrInvalidRequest|normalizeKind' --glob '*.go'; git diff -- request/request.go gateway/handler.go queue/queue.go notifications/notifier.go correlation/identities.go scheduler/jobs.go reporting/report.go storage/requests.go; go build ./... && \
go test ./..."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Normalize"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
7—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
8—✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat request/request_test.go'
9—semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
10—semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
13—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
14—✓ ✓
/bin/zsh -lc "git diff -- request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go queue/queue.go reporting/report.go scheduler/jobs.go storage/requests.go storage/requests_test.go correlation/identities.go go.mod go.sum && \
git status --short -- request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go queue/queue.go reporting/report.go scheduler/jobs.go storage/requests.go storage/requests_test.go correlation/identities.go go.mod go.sum && \
rg -n '\\bRequest\\s*\\{\\s*ID\\b|Request\\.ID|\\.Request\\.ID|r\\.ID' --glob '*.go' ."
15—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.RequestID"
}
16—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Request.Validate"
}
17—✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' . && \
git diff --check"

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency42.95s76.46s+78.0%50.85s96.35s+89.5%
Process Start → First Event0.15s0.14s—0.14s0.16s—
First Event → First Tool Call4.11s5.64s—4.85s4.93s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles611+83.3%812+50.0%
Initial Load / Discovery Turns32-33.3%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)611+83.3%812+50.0%
Output Tokens14091487+5.5%17921802+0.6%
Reasoning / Thinking Tokens341708+107.6%448746+66.5%
Total Input Tokens151094325584+115.5%201803484679+140.2%
Cached Input Tokens136448294912+116.1%183040450560+146.2%
Uncached Input Tokens1464630672+109.4%1876334119+81.8%
Cost0.09260.1778+92.1%0.12070.2298+90.4%
Cached vs Uncached Token Ratio9.32:19.62:1+3.2%9.76:113.21:1+35.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'README*' -g 'PROJECT_BRIEF.md' -g '*.go' | head -120"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc "rg --files -g 'go.mod' -g 'Makefile' -g 'README*' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md'; rg -n '\\bRequest\\b|\\.ID\\b|Normalize|Validate|ErrInvalid' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; cat go.mod; cat request/request.go; rg -n '\\.ID|Request\\{|Normalize|Validate' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat request/request_test.go && \
cat go.mod && \
cat gateway/handler.go && \
cat storage/requests.go && \
git status --short'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
files = ['gateway/handler.go','notifications/notifier.go','request/request.go','queue/queue.go','storage/requests.go','scheduler/jobs.go','correlation/identities.go','reporting/report.go']
for name in files:
    p = Path(name)
    s = p.read_text()
    if name == 'correlation/identities.go':
        s = s.replace('[]string{r.ID, resp.ID, review.ID}', '[]string{r.RequestID, resp.ID, review.ID}')
    elif name == 'storage/requests.go':
        s = s.replace('record.ID == r.ID', 'record.ID == r.RequestID')
    else:
        s = s.replace('r.ID', 'r.RequestID').replace('left.Request.ID', 'left.Request.RequestID').replace('right.Request.ID', 'right.Request.RequestID').replace('job.Request.ID', 'job.Request.RequestID')
    if name == 'request/request.go':
        s = s.replace('\"errors\"\\n', '\"errors\"\\n')
        s = s.replace('ID      string', 'RequestID string')
        s = s.replace('if r.ID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")\\n\\t}', 'if r.RequestID == \"\" {\\n\\t\\treturn ErrInvalidRequest\\n\\t}')
        s = s.replace('type Request struct {', 'var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\ntype Request struct {')
        s = s.replace('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}')
        s = s.replace('if r.ID == id {', 'if r.RequestID == id {')
    p.write_text(s)
PY
rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
for name in ['notifications/notifier.go','reporting/report.go']:
 p=Path(name); s=p.read_text().replace('r.Request.ID','r.Request.RequestID'); p.write_text(s)
PY
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go
go build ./... && \
go test ./..."
✓ ✓
/bin/zsh -lc "git diff --stat && \
git diff -- . ':"'!go.mod'"' ':"'!go.sum'"' && \
rg -n '\\.ID|RequestID' --glob '*.go' ."
6✓ ✓
/bin/zsh -lc "git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go go.mod go.sum && \
rg -n '\\.ID\\b' --glob '*.go' . && \
git diff --check"
✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- docs/SUBOPTIMAL_TOOLS.md tools/benchmark-harness/oracle.go && \
cat request/request.go && \
git diff -- request gateway notifications queue scheduler storage reporting correlation'
7—semedit/semantic_insert_decl (✓ ✓)
{
  "access_modifier": "public",
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
9—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
11—✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n '\\.ID|RequestID' --glob '*.go' . && \
git status --short -- request go.mod go.sum"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README.md' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat go.mod && \
sed -n '1,240p' request/request.go && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize|Validate' --glob '*.go'"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat request/request.go && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize\\(' --glob '*.go' && \
rg -n 'semedit|semantic_' . --glob '*.md' --glob '*.json' --glob '*.yaml' | head -100"
3✓ ✓
/bin/zsh -lc 'cat gateway/handler.go notifications/notifier.go queue/queue.go review/review.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go models/models.go'
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request"
}
4✓ ✓
/bin/zsh -lc 'cat request/request_test.go gateway/handler_test.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
5file_change (✓ ✓)semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "placement": "public_start",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
6✓ ✓
/bin/zsh -lc "gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
rg -n '\\.ID\\b|Request\\{\"|Request\\{' --glob '*.go' && \
git diff --check && \
git diff --stat && \
git diff"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- request/request.go && \
git ls-files request/request.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
8✓ ✓
/bin/zsh -lc 'go build ./... && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "request/request.go",
  "placement": "private_end",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
9—✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|RequestID|normalizeKind|ErrInvalidRequest' --glob '*.go'"
10—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short && \
git diff --'
11—✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short -- request/request.go gateway/handler.go && \
git diff -- request/request.go gateway/handler.go && \
git check-ignore -v request/request.go'
12—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency52.76s98.95s+87.5%62.11s107.41s+72.9%
Process Start → First Event0.13s0.15s—0.14s0.15s—
First Event → First Tool Call5.18s6.09s—5.52s5.33s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles715+114.3%911+22.2%
Initial Load / Discovery Turns43-25.0%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)715+114.3%911+22.2%
Output Tokens19951992-0.2%19531806-7.5%
Reasoning / Thinking Tokens324753+132.4%358996+178.2%
Total Input Tokens175051400614+128.9%247633346482+39.9%
Cached Input Tokens156672350464+123.7%229632316160+37.7%
Uncached Input Tokens1837950150+172.9%1800130322+68.4%
Cost0.11410.2473+116.7%0.13130.1899+44.6%
Cached vs Uncached Token Ratio8.52:16.99:1-18.0%12.76:110.43:1-18.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'AGENTS.md' -g '*.go' | head -100 && \
rg -n 'type Request|func \\(.*Request.*Normalize|func .*Validate|Request\\.ID|\\.ID' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' -g 'README*' | head -60"
2? ✗
/bin/zsh -lc "cat request/request.go && \
cat go.mod && \
rg --files -g 'README*' -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' ."
✓ ✓
/bin/zsh -lc 'rg --files | head -100 && \
cat AGENTS.override.md'
3✓ ✓
/bin/zsh -lc "rg -n 'Request\\{|\\.Request\\.|\\.ID|invalid request|Normalize\\(' --glob '*.go' . && \
rg --files"
✓ ✓
/bin/zsh -lc 'cat docs/identity-model.md && \
cat request/request.go'
4✓ ✓
/bin/zsh -lc "for f in gateway/handler.go notifications/notifier.go request/request.go request/request_test.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go; do printf '\\n--- %s ---\\n' \""'$f"; cat "$f"; done'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
changes = {
'request/request.go': [
('\"errors\"\\n', '\"errors\"\\n'),
('ID      string', 'RequestID string'),
('if r.ID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")\\n\\t}', 'if r.RequestID == \"\" {\\n\\t\\treturn ErrInvalidRequest\\n\\t}'),
('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\nfunc Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}'),
('if r.ID == id {', 'if r.RequestID == id {'),
],
'gateway/handler.go': [('r.ID, err', 'r.RequestID, err')],
'notifications/notifier.go': [('r.ID', 'r.RequestID'), ('r.Request.ID', 'r.Request.RequestID')],
'queue/queue.go': [('r.ID', 'r.RequestID')],
'storage/requests.go': [('record.ID == r.ID', 'record.ID == r.RequestID')],
'scheduler/jobs.go': [('left.Request.ID', 'left.Request.RequestID'), ('right.Request.ID', 'right.Request.RequestID'), ('job.Request.ID', 'job.Request.RequestID'), ('r.ID', 'r.RequestID')],
'correlation/identities.go': [('{r.ID, resp.ID, review.ID}', '{r.RequestID, resp.ID, review.ID}')],
'reporting/report.go': [('r.Request.ID', 'r.Request.RequestID')],
'gateway/handler_test.go': [('request.Request{\"req-1\", \"audit\", nil}', 'request.Request{RequestID: \"req-1\", Kind: \"audit\"}')],
'request/request_test.go': [('import \"testing\"', 'import (\\n\\t\"errors\"\\n\\t\"testing\"\\n)'), ('if err == nil {\\n\\t\\tt.Fatal(\"Validate() accepted an empty request identity\")\\n\\t}', 'if "'!errors.Is(err, ErrInvalidRequest) {'"\\n\\t\\tt.Fatalf(\"Validate() error = %v, want ErrInvalidRequest\", err)\\n\\t}'), ('Request{\"req-7\", \"audit\", nil}', 'Request{RequestID: \"req-7\", Kind: \"audit\"}'), ('[]Request{{\"other\", \"audit\", nil}, want}', '[]Request{{RequestID: \"other\", Kind: \"audit\"}, want}')],
'storage/requests_test.go': [('request.Request{\"req-2\", \"\", nil}', 'request.Request{RequestID: \"req-2\"}')],
}
for name, replacements in changes.items():
    path = Path(name)
    data = path.read_text()
    for old, new in replacements:
        if old not in data:
            raise SystemExit(f'missing expected text in {name}: {old"'!r}'"')
        data = data.replace(old, new)
    fd, tmp = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except:
        try: os.unlink(tmp)
        except FileNotFoundError: pass
        raise
PY
gofmt -w gateway/handler.go gateway/handler_test.go notifications/notifier.go request/request.go request/request_test.go queue/queue.go storage/requests.go storage/requests_test.go scheduler/jobs.go correlation/identities.go reporting/report.go"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "(*Request).Validate"
}
6✓ ✓
/bin/zsh -lc "rg -n '\\.ID|Request\\{["'^'"\\n]*\"req-|type Request struct|normalizeKind|ErrInvalidRequest' --glob '*.go' . && \
git diff -- go.mod go.sum && \
git diff --stat && \
go build ./..."
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Normalize"
}
7✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- request/request.go request/request_test.go gateway/handler.go gateway/handler_test.go notifications/notifier.go queue/queue.go storage/requests.go storage/requests_test.go scheduler/jobs.go correlation/identities.go reporting/report.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
8—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
9—semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
10—semedit/semantic_batch (✓ ✓)
{
  "auto_organize_imports": true,
  "edits": [
    {
      "params": {
        "access_modifier": "public",
        "file": "request/request.go",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "placement": "private_start",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
11—✓ ✓
/bin/zsh -lc "git status --short -- go.mod go.sum && \
rg -n '\\b(Request\\s*\\{["'^}]*'"\\bID:|\\.Request\\.ID|\\bRequest\\s*\\{)' --glob '*.go' ."
12—✓ ✓
/bin/zsh -lc 'git diff --stat && \
git diff -- request/request.go correlation/identities.go gateway/handler.go notifications/notifier.go queue/queue.go reporting/report.go request/request_test.go scheduler/jobs.go storage/requests.go gateway/handler_test.go storage/requests_test.go response review'
13—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.RequestID"
}
14—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request"
}
15—✓ ✓
/bin/zsh -lc 'git status --short && \
cat request/request.go && \
git diff -- request/request.go && \
git show HEAD:request/request.go | head -50 && \
git diff -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat go.mod && \
cat request/request.go && \
rg -n '\\.ID|Request\\{|Normalize|Validate|ErrInvalid' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n '\\b(ID|Normalize|Validate)\\b|\\.ID\\b|ErrInvalidRequest' --glob '*.go' ."
3? ✗
/bin/zsh -lc 'for f in gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go request/request_test.go; do echo ===$f; cat "$f"; done; rg -n '"'request\\.Request|Request\\b|RequestID' --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
4✓ ✓
/bin/zsh -lc "for f in gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go request/request_test.go; do printf '%s\\n' \""'$f"; cat "$f"; done; rg -n '"'request\\.Request|Request\\b|RequestID' --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
5file_change (✓ ✓)semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
6file_change (✓ ✓)semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
7? ✗
/bin/zsh -lc "rg -n '\\.ID|Request\\{' --glob '*.go' . && \
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go && \
git diff -- go.mod go.sum && \
git diff --stat && \
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
8file_change (✓ ✓)semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w correlation/identities.go && \
go build ./... && \
go test ./... && \
git status --short && \
git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go correlation/identities.go'
✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat correlation/identities.go && \
rg -n 'RequestID|\\.ID\\b|ErrInvalidRequest|invalid request|Normalize|normalizeKind' --glob '*.go' ."
10—✓ ✓
/bin/zsh -lc 'git diff --stat && \
git diff -- go.mod go.sum && \
git diff --'
11—✓ ✓
/bin/zsh -lc 'git status --short && \
git ls-files -v request/request.go gateway/handler.go notifications/notifier.go'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency53.77s128.08s+138.2%53.58s67.18s+25.4%
Process Start → First Event0.14s0.25s—0.15s0.15s—
First Event → First Tool Call5.42s4.94s—5.08s6.07s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles715+114.3%990%
Initial Load / Discovery Turns32-33.3%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)715+114.3%990%
Output Tokens20472711+32.4%18161624-10.6%
Reasoning / Thinking Tokens6321531+142.2%398802+101.5%
Total Input Tokens187050595802+218.5%222745342708+53.9%
Cached Input Tokens160768559360+247.9%204288306688+50.1%
Uncached Input Tokens2628236442+38.7%1845736020+95.2%
Cost0.13940.2840+103.7%0.12490.1970+57.8%
Cached vs Uncached Token Ratio6.12:115.35:1+150.9%11.07:18.51:1-23.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200 && \
rg -n '\\bRequest\\b|ErrInvalidRequest|func Normalize|\\.ID\\b' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*request*' -g '*Request*' -g 'README.md' | head -100"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat request/request_test.go && \
cat go.mod && \
rg --files | sort && \
git status --short && \
rg -n 'Request\\.ID|RequestID|\\.ID|Normalize\\(' --glob '"'!go.sum'"' --glob '"'!go.mod'"' ."
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat request/request.go && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize\\(|Validate\\(' --glob '*.go'"
3✓ ✓
/bin/zsh -lc 'cat docs/identity-model.md && \
cat gateway/handler.go queue/queue.go scheduler/jobs.go reporting/report.go storage/requests.go notifications/notifier.go correlation/identities.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
4file_change (✓ ✓)semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "symbol": "Request.ID",
  "to": "RequestID"
}
5✓ ✓
/bin/zsh -lc "gofmt -w request/request.go gateway/handler.go queue/queue.go scheduler/jobs.go reporting/report.go storage/requests.go notifications/notifier.go correlation/identities.go && \
rg -n '\\b(r|left\\.Request|right\\.Request|job\\.Request|entry\\.Value|row\\.Request|record\\.Request)\\.ID\\b|Request\\.ID|\\.Request\\.ID' --glob '*.go' . && \
git diff -- go.mod go.sum && \
go test ./..."
semedit/semantic_replace_body (✓ ✗): replace_body Validate in /Users/alessandro/.codex/worktrees/5ed4eaeb-36d5-4c10-aa76-b12d80abec0e/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-12-request-identity-migration_1790659526092910000/request/request.go: symbol not found
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Validate"
}
6✓ ✓
/bin/zsh -lc "go build ./... && \
git diff -- request/request.go gateway/handler.go queue/queue.go scheduler/jobs.go reporting/report.go storage/requests.go notifications/notifier.go correlation/identities.go && \
git diff -- go.mod go.sum && \
rg -n '\\bRequest\\s*\\{|\\.ID\\b' --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Validate"
}
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
git diff -- request/request.go && \
git ls-files request/request.go go.mod go.sum && \
git status --short --untracked-files=all | head -100'
✓ ✓
/bin/zsh -lc "rg --files | rg 'SUBOPTIMAL_TOOLS|request/request.go|README|PROJECT_BRIEF'"
8—semedit/report_feedback (✓ ✓)
{
  "command": "mcp__semedit__semantic_replace_body",
  "intent": "Replace the Request.Validate method body using the Go semantic editor.",
  "interface": "mcp",
  "manual_touchups": "None yet.",
  "observed_result": "The request failed with a symbol not found error when given the unqualified method name Validate. A follow-up lookup returned the qualified symbol Request.Validate.",
  "parameters": {
    "file": "request/request.go",
    "symbol": "Validate"
  },
  "target_context": "request.Request.Validate",
  "unexpected_reason": "The method existed and had been found in the selected source file, but the mutation required a receiver-qualified selector."
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
10—semedit/semantic_insert_decl (✓ ✓)
{
  "access_modifier": "public",
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
11—semedit/semantic_lookup (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
13—semedit/semantic_insert_function (✓ ✓)
{
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
14—✓ ✓
/bin/zsh -lc "git diff -- && \
git status --short && \
rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go'"
15—✓ ✓
/bin/zsh -lc 'git diff -- request/request.go gateway/handler.go && \
git status --short -- request/request.go gateway/handler.go request/request_test.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat go.mod && \
cat request/request.go && \
rg -n '\\.ID|ErrInvalid|Normalize|Validate' --glob '*.go' . && \
rg --files"
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; cat go.mod; rg -n 'Request|\\.ID|Normalize|Validate' --glob '*.go' | head -200"
3✓ ✓
/bin/zsh -lc 'cat models/models.go request/request_test.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go docs/identity-model.md && \
git status --short'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
4✓ ✓
/bin/zsh -lc "cat review/review.go response/response.go && \
rg -n 'request\\.Request\\{|Request\\{' --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p=Path('request/request.go')
s=p.read_text()
s=s.replace('\"errors\"\\n\\t\"strings\"','\"errors\"\\n\\t\"strings\"')
s=s.replace('type Request struct {\\n\\tID      string','var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\ntype Request struct {\\n\\tRequestID string')
s=s.replace('if r.ID == \"\" {\\n\\t\\treturn errors.New(\"invalid request\")','if r.RequestID == \"\" {\\n\\t\\treturn ErrInvalidRequest')
s=s.replace('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}')
s=s.replace('if r.ID == id {','if r.RequestID == id {')
p.write_text(s)
changes={
'gateway/handler.go': [('r.ID','r.RequestID')],
'notifications/notifier.go': [('r.ID','r.RequestID'),('r.Request.ID','r.Request.RequestID')],
'queue/queue.go': [('r.ID','r.RequestID')],
'storage/requests.go': [('record.ID == r.ID','record.ID == r.RequestID')],
'scheduler/jobs.go': [('left.Request.ID','left.Request.RequestID'),('right.Request.ID','right.Request.RequestID'),('job.Request.ID','job.Request.RequestID'),('r.ID','r.RequestID')],
'correlation/identities.go': [('{r.ID, resp.ID, review.ID}','{r.RequestID, resp.ID, review.ID}')],
'reporting/report.go': [('r.Request.ID','r.Request.RequestID')],
}
for name,repls in changes.items():
 p=Path(name); s=p.read_text()
 for a,b in repls:s=s.replace(a,b)
 p.write_text(s)
PY
"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
6✓ ✓
/bin/zsh -lc "git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
rg -n '\\b(r|left\\.Request|right\\.Request|job\\.Request)\\.ID|Request\\.ID' --glob '*.go' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize"
}
7✓ ✓
/bin/zsh -lc "sed -n '1,80p' request/request.go && \
git status --short && \
git rev-parse --show-toplevel"
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "request/request.go",
        "placement": "private_end",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
8✓ ✓
/bin/zsh -lc 'gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
go test ./...'
✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff -- && \
rg -n 'Request\\{.*|Request\\.ID|Request\\.RequestID|r\\.ID|Request\\.ID|\\.Request\\.ID' --glob '*.go'"
9✓ ✓
/bin/zsh -lc "go build ./... && \
rg -n '\\.ID|ErrInvalidRequest|normalizeKind' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "sed -n '1,120p' request/request.go && \
rg -n '\\b(ID|RequestID)\\b' --glob '*.go' | head -200 && \
git diff --name-only -- request/request.go go.mod go.sum"

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency48.02s99.69s+107.6%51.66s94.29s+82.5%
Process Start → First Event0.17s0.19s—0.14s0.16s—
First Event → First Tool Call4.53s4.58s—4.70s4.20s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles814+75.0%614+133.3%
Initial Load / Discovery Turns34+33.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)814+75.0%614+133.3%
Output Tokens16921855+9.6%18002218+23.2%
Reasoning / Thinking Tokens415933+124.8%6471018+57.3%
Total Input Tokens195140296893+52.1%158101471365+198.1%
Cached Input Tokens175872264448+50.4%140544419584+198.5%
Uncached Input Tokens1926832445+68.4%1755751781+194.9%
Cost0.11850.1821+53.7%0.10960.2748+150.7%
Cached vs Uncached Token Ratio9.13:18.15:1-10.7%8.01:18.10:1+1.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*request*' -g 'README.md' | head -120"
2✓ ✓
/bin/zsh -lc "cat go.mod && \
cat request/request.go && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize|Validate|ErrInvalid' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat request/request_test.go'
3✓ ✓
/bin/zsh -lc 'cat request/request_test.go && \
cat gateway/handler.go && \
cat correlation/identities.go && \
cat review/review.go && \
cat response/response.go && \
cat storage/requests.go'
✓ ✓
/bin/zsh -lc "rg -n '\\bRequest\\b|\\.ID\\b|Validate\\(|Normalize\\(' --glob '*.go'"
4file_change (✓ ✓)✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README.md' -g 'Makefile' -g '*.go' | head -160"
5? ✗
/bin/zsh -lc "rg -n '\\b(r|left\\.Request|right\\.Request|job\\.Request|record\\.Request|.*Request)\\.ID\\b|\\.Request\\.ID\\b' --glob '*.go'; rg --files -g 'Makefile' -g 'README*' -g 'go.sum' && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
6✓ ✓
/bin/zsh -lc 'go build ./...'
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
7✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "group": "standalone",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
8✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "(*Request).Validate"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "placement": "private_end",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
11—✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' && \
git diff --"
12—semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.RequestID"
}
13—semedit/semantic_lookup (✓ ✓)
{
  "symbol": "normalizeKind"
}
14—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
git diff -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc "ls -la && \
cat go.mod && \
rg -n 'Request|\\.ID|Normalize|Validate' --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat request/request.go'
3✓ ✓
/bin/zsh -lc "cat request/request.go request/request_test.go && \
rg -n '\\brequest\\.Request\\s*\\{|\\bRequest\\s*\\{' --glob '*.go' . && \
rg -n '\\.ID' --glob '*.go' ."
semedit/semantic_outline (✓ ✓)
{
  "include_tests": true,
  "path": "request"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
files = {
    'request/request.go': [('ID      string', 'RequestID string'), ('r.ID == \"\"', 'r.RequestID == \"\"'), ('return errors.New(\"invalid request\")', 'return fmt.Errorf(\"%w: request ID is empty\", ErrInvalidRequest)'), ('kind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind', 'return normalizeKind(kind)'), ('if r.ID == id', 'if r.RequestID == id')],
    'gateway/handler.go': [('r.ID, err', 'r.RequestID, err')],
    'notifications/notifier.go': [('r.ID', 'r.RequestID'), ('r.Request.ID', 'r.Request.RequestID')],
    'queue/queue.go': [('r.ID', 'r.RequestID')],
    'storage/requests.go': [('record.ID == r.ID', 'record.ID == r.RequestID')],
    'scheduler/jobs.go': [('left.Request.ID', 'left.Request.RequestID'), ('right.Request.ID', 'right.Request.RequestID'), ('job.Request.ID', 'job.Request.RequestID'), ('r.ID', 'r.RequestID')],
    'correlation/identities.go': [('r.ID', 'r.RequestID')],
    'reporting/report.go': [('r.Request.ID', 'r.Request.RequestID')],
}
for name, pairs in files.items():
    p = Path(name)
    s = p.read_text()
    for old, new in pairs:
        if old not in s:
            raise SystemExit(f'missing {old"'!r} in {name}'"')
        s = s.replace(old, new)
    if name == 'request/request.go':
        s = s.replace('\"errors\"', '\"errors\"\\n\\t\"fmt\"')
        s = s.replace('type Request struct {', 'var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\ntype Request struct {')
        s = s.replace('func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}')
    p.write_text(s)
PY
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go
"
semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID"
}
5✓ ✓
/bin/zsh -lc "git diff --check && \
git diff -- && \
rg -n '\\br\\.ID|Request\\.ID|left\\.Request\\.ID|right\\.Request\\.ID|job\\.Request\\.ID' --glob '*.go' . && \
git status --short"
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
6✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
go build ./... && \
go test ./...'
semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_insert_decl", "status": "error", "error": "calculate declaration offset: section placement violation: strategy \"private_start\", in /Users/alessandro/.codex/worktrees/5ed4eaeb-36d5-4c10-aa76-b12d80abec0e/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-12-request-identity-migration_1790660383507592000/request/request.go" } ], "final_diff": "" }
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "placement": "private_start",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "request/request.go",
        "placement": "private_end",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
7—✓ ✓
/bin/zsh -lc "rg --files -g 'SUBOPTIMAL_TOOLS.md' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'; git diff --stat && \
cat request/request.go"
8—? ✗
/bin/zsh -lc 'cat docs/SUBOPTIMAL_TOOLS.md && \
git status --short && \
git diff -- request/request.go docs/SUBOPTIMAL_TOOLS.md'
9—✓ ✓
/bin/zsh -lc 'pwd; ls -la; git status --short'
10—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "placement": "public_end",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "request/request.go",
        "placement": "private_end",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
11—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request"
}
12—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Normalize"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "ErrInvalidRequest"
}
14—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.RequestID"
}

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency46.03s112.50s+144.4%47.22s87.43s+85.2%
Process Start → First Event0.13s0.13s—0.15s0.14s—
First Event → First Tool Call4.76s7.46s—5.61s5.17s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles717+142.9%713+85.7%
Initial Load / Discovery Turns440%43-25.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)717+142.9%713+85.7%
Output Tokens16792633+56.8%14561763+21.1%
Reasoning / Thinking Tokens3581176+228.5%376701+86.4%
Total Input Tokens176910509453+188.0%173070464471+168.4%
Cached Input Tokens159744453632+184.0%142336430336+202.3%
Uncached Input Tokens1716655821+225.2%3073434135+11.1%
Cost0.10830.3006+177.5%0.13530.2237+65.3%
Cached vs Uncached Token Ratio9.31:18.13:1-12.7%4.63:112.61:1+172.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | head -80 && \
rg -n \"type Request|func \\(["'^)]*Request'"\\) (Validate|Normalize)|\\.ID\\b|ErrInvalidRequest\" --glob '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat docs/identity-model.md && \
cat request/request_test.go && \
rg -n \"request\\.Request|Request\\{\" --glob '*.go'"
? ✗
/bin/zsh -lc "cat PROJECT_BRIEF.md && \
cat go.mod && \
rg -n 'Request|\\.ID|Validate|Normalize' --glob '*.go'"
3? ✗
/bin/zsh -lc 'for f in gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go reporting/report.go correlation/identities.go storage/requests.go response/response.go review/review.go; do echo ===$f; cat "$f"; done; rg -n "'"\\.ID\\b|ID:\" --glob '*.go'"
✓ ✓
/bin/zsh -lc "rg --files -g '*.md' -g 'AGENTS*' -g 'go.mod' | head -50 && \
cat go.mod && \
rg -n 'Request|\\.ID|Validate|Normalize' --glob '*.go'"
4✓ ✓
/bin/zsh -lc "cat gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go reporting/report.go correlation/identities.go storage/requests.go response/response.go review/review.go; rg -n \"\\.ID\\b|ID:\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat docs/identity-model.md && \
cat request/request.go && \
cat request/request_test.go'
5file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
6✓ ✓
/bin/zsh -lc "ls -la && \
rg -n \"Request\\.ID|r\\.ID|RequestID|func normalizeKind|ErrInvalidRequest\" --glob '*.go'"
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go reporting/report.go correlation/identities.go storage/requests.go && \
go build ./... && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.Validate"
}
8—semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Normalize"
}
9—semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request"
}
10—semedit/semantic_insert_decl (✓ ✗): validate declaration snippet: syntax error: expected declaration, found ErrInvalidRequest
{
  "access_modifier": "public",
  "file": "request/request.go",
  "source": "ErrInvalidRequest = errors.New(\"invalid request\")"
}
11—✓ ✓
/bin/zsh -lc 'ls docs && \
cat docs/SUBOPTIMAL_TOOLS.md 2>/dev/null || \
true'
12—✓ ✓
/bin/zsh -lc "python3 -c 'import os,tempfile,time; p=\"docs/SUBOPTIMAL_TOOLS.md\"; d=os.path.dirname(p); fd,t=tempfile.mkstemp(dir=d,prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(\"# Suboptimal semantic tool behavior\\n\\n- Tool: "'`semantic_insert_decl`'"\\n- Target: "'`request/request.go`'"\\n- Failure: rejected "'`ErrInvalidRequest = ...` because it expected a declaration keyword and reported `expected declaration`.'"\\n- Workaround: retry with a complete "'`var ErrInvalidRequest = ...` declaration.'"\\n- Root cause: the tool accepts declaration source, not a bare declaration spec.\\n\"); f.flush(); os.fsync(f.fileno()); os.utime(t,(time.time()+1,time.time()+1)); f.close(); os.replace(t,p); fd=os.open(d,os.O_RDONLY); os.fsync(fd); os.close(fd)'"
13—semedit/semantic_insert_decl (✓ ✓)
{
  "access_modifier": "public",
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
14—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\treturn ErrInvalidRequest\n}\nreturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
15—✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff -- request/request.go docs/SUBOPTIMAL_TOOLS.md && \
rg -n '\\b(r|left\\.Request|right\\.Request|job\\.Request|entry\\.Value|record\\.Request)\\.ID\\b|Request\\{["'^}]*ID:'"' --glob '*.go'"
16—semedit/semantic_find_references (✓ ✓)
{
  "symbol": "Request.RequestID"
}
17—✓ ✓
/bin/zsh -lc "cat request/request.go && \
cat docs/SUBOPTIMAL_TOOLS.md && \
rg -n 'RequestID|\\.ID' --glob '*.go'"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'Makefile' -g '*.go' -g 'go.mod' -g 'go.sum' | head -100"
2✓ ✓
/bin/zsh -lc "cat go.mod && \
rg -n \"\\.ID\\b|\\bID\\b|func .*Normalize|Validate|ErrInvalidRequest\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat request/request.go && \
git status --short && \
rg -n '\\b(ID|RequestID)\\b|Normalize|Validate' --glob '*.go'"
3? ✗
/bin/zsh -lc "cat request/request.go && \
cat request/request_test.go && \
rg -n 'request\\.Request\\{|Request\\{' --glob '*.go' . && \
rg --files -g 'README*' -g 'Makefile' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc 'cat request/request_test.go; cat gateway/handler.go; cat notifications/notifier.go; cat queue/queue.go'
4✓ ✓
/bin/zsh -lc 'for f in gateway/handler.go notifications/notifier.go storage/requests.go reporting/report.go scheduler/jobs.go queue/queue.go correlation/identities.go; do echo ---$f; cat "$f"; done'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p=Path('request/request.go')
s=p.read_text()
s=s.replace('type Request struct {\\n\\tID      string', 'var ErrInvalidRequest = errors.New(\"invalid request\")\\n\\ntype Request struct {\\n\\tRequestID string')
s=s.replace('r.ID', 'r.RequestID')
s=s.replace('return errors.New(\"invalid request\")', 'return ErrInvalidRequest')
s=s.replace('func Normalize(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}', 'func Normalize(kind string) string {\\n\\treturn normalizeKind(kind)\\n}\\n\\nfunc normalizeKind(kind string) string {\\n\\tkind = strings.TrimSpace(kind)\\n\\tkind = strings.ToLower(kind)\\n\\treturn kind\\n}')
p.write_text(s)
files={
'gateway/handler.go':[('r.ID','r.RequestID')],
'notifications/notifier.go':[('r.ID','r.RequestID'),('r.Request.ID','r.Request.RequestID')],
'storage/requests.go':[('record.ID == r.ID','record.ID == r.RequestID')],
'reporting/report.go':[('r.Request.ID','r.Request.RequestID')],
'scheduler/jobs.go':[('left.Request.ID','left.Request.RequestID'),('right.Request.ID','right.Request.RequestID'),('job.Request.ID','job.Request.RequestID'),('r.ID','r.RequestID')],
'queue/queue.go':[('r.ID','r.RequestID')],
'correlation/identities.go':[('r.ID','r.RequestID')],
}
for name,repls in files.items():
 p=Path(name); s=p.read_text()
 for a,b in repls: s=s.replace(a,b)
 p.write_text(s)
PY
rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w request/request.go gateway/handler.go notifications/notifier.go storage/requests.go reporting/report.go scheduler/jobs.go queue/queue.go correlation/identities.go && \
go build ./... && \
go test ./...'
semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff -- request/request.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
  "file": "request/request.go",
  "symbol": "(*Request).Validate"
}
8—semedit/semantic_insert_function (✓ ✓)
{
  "file": "request/request.go",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
10—✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n 'request\\.Request|\\.Request\\.(ID|RequestID)|\\br\\.ID\\b|\\bID:' --glob '*.go'"
11—✓ ✓
/bin/zsh -lc 'cat gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go request/request_test.go'
12—✓ ✓
/bin/zsh -lc "rg -n '\\br\\.ID\\b|Request\\.(ID)\\b|\\.Request\\.ID\\b|Request\\{["'^'"\\n]*\\bID:' --glob '*.go' && \
git diff -- go.mod go.sum; git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go reporting/report.go request/request_test.go"
13—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff --stat && \
git diff --check && \
git ls-files request/request.go'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency51.99s74.14s+42.6%58.33s81.72s+40.1%
Process Start → First Event0.14s0.15s—0.15s0.14s—
First Event → First Tool Call5.54s5.53s—6.04s4.78s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles811+37.5%712+71.4%
Initial Load / Discovery Turns330%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)811+37.5%712+71.4%
Output Tokens17061290-24.4%19041588-16.6%
Reasoning / Thinking Tokens330403+22.1%313806+157.5%
Total Input Tokens192616417167+116.6%178293307656+72.6%
Cached Input Tokens175872386560+119.8%160768275712+71.5%
Uncached Input Tokens1674430607+82.8%1752531944+82.3%
Cost0.11130.1943+74.6%0.11170.1787+60.0%
Cached vs Uncached Token Ratio10.50:112.63:1+20.2%9.17:18.63:1-5.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | head -80 && \
rg -n 'type Request|func \\(.*Request.*\\) Validate|func Normalize|\\.ID\\b|ErrInvalidRequest' --glob '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' -g 'Makefile' -g 'README*' | head -100"
2? ✗
/bin/zsh -lc "cat request/request.go && \
rg -n 'Request\\{|\\.Request\\.|Request\\)' --glob '*.go' && \
cat docs/identity-model.md && \
rg --files -g 'Makefile' -g 'go.work' -g 'AGENTS.md'"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat PROJECT_BRIEF.md 2>/dev/null || \
true && \
rg -n 'Request\\.ID|\\.ID|func .*Normalize|type Request|func .*Validate' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc 'cat request/request_test.go && \
cat go.mod && \
ls'
✓ ✓
/bin/zsh -lc 'cat request/request.go && \
cat go.mod'
4file_change (✓ ✓)semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
5file_change (✓ ✓)semedit/semantic_insert_decl (✓ ✓)
{
  "access_modifier": "public",
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
6✓ ✓
/bin/zsh -lc "rg -n '\\b(r|left\\.Request|right\\.Request|job\\.Request|record|resp|review)\\.ID\\b|Request\\{["'^'"\\n]*' --glob '*.go' && \
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go scheduler/jobs.go storage/requests.go reporting/report.go correlation/identities.go && \
go build ./... && \
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"validate request: %w\", ErrInvalidRequest)\n}\nreturn nil",
  "file": "request/request.go",
  "symbol": "(*Request).Validate"
}
7file_change (✓ ✓)semedit/semantic_replace_body (✓ ✓)
{
  "body": "return normalizeKind(kind)",
  "file": "request/request.go",
  "symbol": "Normalize"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w request/request_test.go && \
go build ./... && \
go test ./... && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "request/request.go",
  "placement": "after_symbol",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
  "target_symbol": "Normalize"
}
9—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff'
10—✓ ✓
/bin/zsh -lc 'cat request/request.go; git status --short -- request/request.go go.mod go.sum; git rev-parse --show-toplevel; pwd'
11—✓ ✓
/bin/zsh -lc "rg -n 'Request\\.ID|RequestID|\\.ID' --glob '*.go' ."
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File request/request.go modified (34 lines -> 40 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg -n \"type Request|func \\(.*Request.*\\) Validate|func Normalize|ErrInvalidRequest|\\.ID\\b\" --glob '*.go' && \
rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat request/request.go && \
rg -n \"Request\\{|\\.Request\\.ID|request\\.Request|Request\\.ID|\\.ID\" --glob '*.go' && \
rg --files"
✓ ✓
/bin/zsh -lc "rg --files | sort | head -120 && \
rg -n 'func .*Normalize|type Request|\\.ID\\b|ErrInvalid|Validate\\('"
3✓ ✓
/bin/zsh -lc 'cat request/request_test.go && \
cat gateway/handler.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request"
}
4file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Request.ID"
}
5✓ ✓
/bin/zsh -lc "rg -n \"\\.ID\\b|Request\\{\" --glob '*.go' && \
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go gateway/handler_test.go request/request_test.go storage/requests_test.go && \
go build ./... && \
go test ./..."
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Normalize"
}
6✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff --'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.Validate"
}
7✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=no -- request gateway notifications queue storage scheduler correlation reporting && \
git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize"
}
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID"
}
9—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Request.ID",
  "to": "RequestID"
}
10—semedit/semantic_insert_decl (✓ ✓)
{
  "file": "request/request.go",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
11—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "return normalizeKind(kind)",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "access_modifier": "private",
        "file": "request/request.go",
        "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}"
      },
      "tool": "semantic_insert_function"
    }
  ]
}
12—✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' && \
git diff -- go.mod go.sum && \
git status --short && \
git diff --stat"
Benchmark run task-12-strengthened-readwrite-20260929Benchmark run task-all-gpt-6-luna-medium-prescriptive-repeats10-20260928T164359Z