Benchmark run luna-medium-staged-followup-r10-20260929T054733Z

This page contains exactly the publishable benchmark observations recorded in run luna-medium-staged-followup-r10-20260929T054733Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
4. Level 4 (Test Suite): Unit and integration test suite execution.
Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: `task-13-rename-followup-scale`

Target: `codex/gpt-6-luna/medium`

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	49.65s	262.95s	+429.6%	—	—	—
Process Start → First Event	0.25s	0.19s	—	—	—	—
First Event → First Tool Call	4.84s	6.76s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	41	+485.7%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	41	+485.7%	—	—	—
Output Tokens	2995	5863	+95.8%	—	—	—
Reasoning / Thinking Tokens	320	621	+94.1%	—	—	—
Total Input Tokens	809946	2814644	+247.5%	—	—	—
Cached Input Tokens	743424	2674432	+259.7%	—	—	—
Uncached Input Tokens	66522	140212	+110.8%	—	—	—
Cost	0.3936	1.1002	+179.5%	—	—	—
Cached vs Uncached Token Ratio	11.18:1	19.07:1	+70.7%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'README.md' | head -60"
/bin/zsh -lc "cat rename-plan.md && \
rg --files | head -100 && \
rg -n 'Processor|methodname|TODO' --glob '*.go' ."
{
  "language": "go",
  "symbol": "LoadInput"
}
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod pipeline/workflow_test.go && \
git status --short'
/bin/zsh -lc "rg -n 'ReadRequest|LoadInput' pipeline/workflow.go && \
git diff --no-index /dev/null pipeline/workflow.go"
{
  "language": "go",
  "symbol": "CheckInput"
}
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "NormalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "MapFields"
}
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "language": "go",
  "symbol": "StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "language": "go",
  "symbol": "CountItems"
}
{
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "language": "go",
  "symbol": "MergeParts"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "SaveResult"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ git status --short && \ cat rename-plan.md && \ rg -n 'Processor\|<method-name>' --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'README.md' \| head -60"`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go'`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files \| head -100 && \ rg -n 'Processor\|methodname\|TODO' --glob '*.go' ."`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"LoadInput\")==2; p.write_text(s.replace(\"LoadInput\", \"ReadRequest\"))' && \ git diff -- pipeline/workflow.go && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"CheckInput\")==2; p.write_text(s.replace(\"CheckInput\", \"ValidateRequest\"))' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"NormalizeValue\")==2 and s.count(\"MapFields\")==2; s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=[(\"RouteValue\",\"DispatchValue\"),(\"StampEvent\",\"AnnotateEvent\"),(\"CountItems\",\"TallyItems\"),(\"MergeParts\",\"CombineParts\")]; assert all(s.count(a)==2 for a,b in pairs); [None for a,b in pairs]; for_dummy=0 for old,new in pairs: s=s.replace(old,new) p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod pipeline/workflow_test.go && \ git status --short'`
7	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=[(\"FilterNoise\",\"PruneNoise\"),(\"SortRecords\",\"OrderRecords\"),(\"EncodePayload\",\"MarshalPayload\"),(\"DecodeReply\",\"ParseReply\"),(\"OpenChannel\",\"ConnectChannel\"),(\"CloseChannel\",\"ReleaseChannel\"),(\"FlushBuffer\",\"DrainBuffer\"),(\"SaveResult\",\"PersistResult\")]; assert all(s.count(a)==2 for a,b in pairs) for old,new in pairs: s=s.replace(old,new) p.write_text(s)' && \ go test ./..."	? ✗ `/bin/zsh -lc "rg -n 'ReadRequest\|LoadInput' pipeline/workflow.go && \ git diff --no-index /dev/null pipeline/workflow.go"`
8	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CheckInput" }`
9	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
10	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
11	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "NormalizeValue" }`
12	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MapFields" }`
14	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
15	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "RouteValue" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "StampEvent" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
20	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CountItems" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MergeParts" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
24	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FilterNoise" }`
26	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SortRecords" }`
28	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "EncodePayload" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "DecodeReply" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "OpenChannel" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
35	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CloseChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
37	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FlushBuffer" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
39	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SaveResult" }`
40	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
41	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	50.84s	265.10s	+421.5%	—	—	—
Process Start → First Event	0.15s	0.12s	—	—	—	—
First Event → First Tool Call	4.47s	7.05s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	8	38	+375.0%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	8	38	+375.0%	—	—	—
Output Tokens	3663	4710	+28.6%	—	—	—
Reasoning / Thinking Tokens	500	0	-100.0%	—	—	—
Total Input Tokens	922853	2870760	+211.1%	—	—	—
Cached Input Tokens	850688	2727680	+220.6%	—	—	—
Uncached Input Tokens	72165	143080	+98.3%	—	—	—
Cost	0.4451	1.0985	+146.8%	—	—	—
Cached vs Uncached Token Ratio	11.79:1	19.06:1	+61.7%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -100"
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md go.mod && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ git status --short && \ cat rename-plan.md && \ rg -n \"Processor\|method\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ cat pipeline/workflow_test.go && \ rg -n \"LoadInput\|ReadRequest\" . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); p.write_text(s)' && \ go test ./... && \ git diff -- pipeline/workflow.go && \ git status --short"`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc 'rg -n "ReadRequest\|LoadInput" pipeline/workflow.go && \ git diff --no-ext-diff -- pipeline/workflow.go && \ git ls-files -v pipeline/workflow.go'`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md go.mod && \ go test ./...'`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text().replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
7	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"RouteValue\", \"DispatchValue\"), (\"StampEvent\", \"AnnotateEvent\"), (\"CountItems\", \"TallyItems\"), (\"MergeParts\", \"CombineParts\")); for old, new in pairs: s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"FilterNoise\", \"PruneNoise\"), (\"SortRecords\", \"OrderRecords\"), (\"EncodePayload\", \"MarshalPayload\"), (\"DecodeReply\", \"ParseReply\"), (\"OpenChannel\", \"ConnectChannel\"), (\"CloseChannel\", \"ReleaseChannel\"), (\"FlushBuffer\", \"DrainBuffer\"), (\"SaveResult\", \"PersistResult\")); for old, new in pairs: s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
9	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
10	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
11	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
12	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
14	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
16	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
19	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
21	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
25	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
27	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
29	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
32	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
34	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
36	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
38	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	86.88s	261.22s	+200.7%	—	—	—
Process Start → First Event	0.24s	0.23s	—	—	—	—
First Event → First Tool Call	5.40s	5.77s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	16	39	+143.8%	—	—	—
Initial Load / Discovery Turns	4	0	-100.0%	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	16	39	+143.8%	—	—	—
Output Tokens	7179	5414	-24.6%	—	—	—
Reasoning / Thinking Tokens	0	220	—	—	—	—
Total Input Tokens	1358860	3346094	+146.2%	—	—	—
Cached Input Tokens	1273600	3173376	+149.2%	—	—	—
Uncached Input Tokens	85260	172718	+102.6%	—	—	—
Cost	0.6213	1.2956	+108.5%	—	—	—
Cached vs Uncached Token Ratio	14.94:1	18.37:1	+23.0%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -80"
/bin/zsh -lc 'cat rename-plan.md && \
cat AGENTS.override.md && \
rg --files | head -80'
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md go.mod && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|func \\(.Processor\\)\" --glob '.go'"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'rename-plan.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ rg -n \"LoadInput\" . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md && \ cat AGENTS.override.md && \ rg --files \| head -80'`
3	`file_change` (✓ ✓)	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
4	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "rg -n \"CheckInput\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md go.mod && \ go test ./...'`
6	`file_change` (✓ ✓)	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
7	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
8	✓ ✓ `/bin/zsh -lc "rg -n \"NormalizeValue\|MapFields\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
9	`file_change` (✓ ✓)	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
10	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
11	✓ ✓ `/bin/zsh -lc "rg -n \"RouteValue\|StampEvent\|CountItems\|MergeParts\" --glob '*.go' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
12	`file_change` (✓ ✓)	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
13	✓ ✓ `/bin/zsh -lc 'go test ./...'`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
14	✓ ✓ `/bin/zsh -lc "rg -n \"FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult\" --glob '*.go' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
15	`file_change` (✓ ✓)	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
16	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
20	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
22	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
24	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
26	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
28	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
35	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
37	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
39	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	56.70s	235.60s	+315.5%	—	—	—
Process Start → First Event	0.23s	0.14s	—	—	—	—
First Event → First Tool Call	5.48s	7.99s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	8	42	+425.0%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	8	42	+425.0%	—	—	—
Output Tokens	4746	5074	+6.9%	—	—	—
Reasoning / Thinking Tokens	275	36	-86.9%	—	—	—
Total Input Tokens	918941	3107171	+238.1%	—	—	—
Cached Input Tokens	850688	2917888	+243.0%	—	—	—
Uncached Input Tokens	68253	189283	+177.3%	—	—	—
Cost	0.4461	1.2666	+183.9%	—	—	—
Cached vs Uncached Token Ratio	12.46:1	15.42:1	+23.7%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc 'cat rename-plan.md'
{
  "language": "go",
  "symbol": "LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- go.mod rename-plan.md && \
git status --short && \
git diff --'
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "NormalizeValue"
}
{
  "language": "go",
  "symbol": "MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "RouteValue"
}
{
  "language": "go",
  "symbol": "StampEvent"
}
{
  "language": "go",
  "symbol": "CountItems"
}
{
  "language": "go",
  "symbol": "MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "SortRecords"
}
{
  "language": "go",
  "symbol": "FilterNoise"
}
{
  "language": "go",
  "symbol": "EncodePayload"
}
{
  "language": "go",
  "symbol": "DecodeReply"
}
{
  "language": "go",
  "symbol": "OpenChannel"
}
{
  "language": "go",
  "symbol": "CloseChannel"
}
{
  "language": "go",
  "symbol": "FlushBuffer"
}
{
  "language": "go",
  "symbol": "SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|Rename\" --glob '*.go'"`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md'`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go && \ git status --short'`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); p.write_text(s)'"`	`semedit/semantic_inspect_symbol` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput" }`
4	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'git diff -- go.mod rename-plan.md && \ git status --short && \ git diff --'`
7	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"RouteValue\", \"DispatchValue\").replace(\"StampEvent\", \"AnnotateEvent\").replace(\"CountItems\", \"TallyItems\").replace(\"MergeParts\", \"CombineParts\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"FilterNoise\", \"PruneNoise\"), (\"SortRecords\", \"OrderRecords\"), (\"EncodePayload\", \"MarshalPayload\"), (\"DecodeReply\", \"ParseReply\"), (\"OpenChannel\", \"ConnectChannel\"), (\"CloseChannel\", \"ReleaseChannel\"), (\"FlushBuffer\", \"DrainBuffer\"), (\"SaveResult\", \"PersistResult\")); for old, new in pairs: s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CheckInput" }`
9	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput" }`
10	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
11	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
12	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "NormalizeValue" }`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MapFields" }`
14	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
15	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
16	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "RouteValue" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "StampEvent" }`
19	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CountItems" }`
20	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MergeParts" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
22	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
24	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
25	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SortRecords" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FilterNoise" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "EncodePayload" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "DecodeReply" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "OpenChannel" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CloseChannel" }`
32	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FlushBuffer" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SaveResult" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
39	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
40	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
41	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
42	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	73.70s	225.54s	+206.0%	—	—	—
Process Start → First Event	0.13s	0.14s	—	—	—	—
First Event → First Tool Call	4.79s	5.71s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	14	39	+178.6%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	14	39	+178.6%	—	—	—
Output Tokens	6743	5308	-21.3%	—	—	—
Reasoning / Thinking Tokens	455	605	+33.0%	—	—	—
Total Input Tokens	1329338	2410813	+81.4%	—	—	—
Cached Input Tokens	1242880	2257664	+81.6%	—	—	—
Uncached Input Tokens	86458	153149	+77.1%	—	—	—
Cost	0.6168	1.0212	+65.6%	—	—	—
Cached vs Uncached Token Ratio	14.38:1	14.74:1	+2.5%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "cat rename-plan.md && \
printf '\\n--- files ---\\n' && \
rg --files -g 'AGENTS.md' -g '*.go' | head -80"
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md && \
go test ./...'
/bin/zsh -lc "git status --short && \
rg -n 'LoadInput|ReadRequest' pipeline/workflow.go pipeline/workflow_test.go"
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ git status --short && \ rg --files -g 'AGENTS.md' -g 'AGENTS.override.md'"`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ printf '\\n--- files ---\\n' && \ rg --files -g 'AGENTS.md' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "rg -n '\\bLoadInput\\b\|func \\(.Processor.\\) ' --glob '*.go'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "rg -n '\\bLoadInput\\b' . --glob '"'!rename-plan.md'"'"`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); p.write_text(s)'"`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md && \ go test ./...'`
5	✓ ✓ `/bin/zsh -lc 'go test ./...'`	✓ ✓ `/bin/zsh -lc "git status --short && \ rg -n 'LoadInput\|ReadRequest' pipeline/workflow.go pipeline/workflow_test.go"`
6	✓ ✓ `/bin/zsh -lc "git diff -- pipeline/workflow.go && \ rg -n '\\bLoadInput\\b\|\\bReadRequest\\b' --glob '*.go'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
7	✓ ✓ `/bin/zsh -lc "rg -n '\\bCheckInput\\b' . --glob '"'!rename-plan.md'"' --glob '"'!*/_test.go'"'"`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
8	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
9	✓ ✓ `/bin/zsh -lc "rg -n '\\b(NormalizeValue\|MapFields)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!*/_test.go'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
10	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
11	✓ ✓ `/bin/zsh -lc "rg -n '\\b(RouteValue\|StampEvent\|CountItems\|MergeParts)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!*/_test.go'"'"`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
12	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); for old, new in ((\"RouteValue\", \"DispatchValue\"), (\"StampEvent\", \"AnnotateEvent\"), (\"CountItems\", \"TallyItems\"), (\"MergeParts\", \"CombineParts\")): s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
13	✓ ✓ `/bin/zsh -lc "rg -n '\\b(FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!*/_test.go'"'"`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
14	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); for old, new in ((\"FilterNoise\", \"PruneNoise\"), (\"SortRecords\", \"OrderRecords\"), (\"EncodePayload\", \"MarshalPayload\"), (\"DecodeReply\", \"ParseReply\"), (\"OpenChannel\", \"ConnectChannel\"), (\"CloseChannel\", \"ReleaseChannel\"), (\"FlushBuffer\", \"DrainBuffer\"), (\"SaveResult\", \"PersistResult\")): s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
22	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
39	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	60.08s	233.85s	+289.3%	—	—	—
Process Start → First Event	0.13s	0.14s	—	—	—	—
First Event → First Tool Call	5.24s	6.03s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	8	38	+375.0%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	8	38	+375.0%	—	—	—
Output Tokens	4753	3956	-16.8%	—	—	—
Reasoning / Thinking Tokens	480	0	-100.0%	—	—	—
Total Input Tokens	913503	2864957	+213.6%	—	—	—
Cached Input Tokens	840448	2721536	+223.8%	—	—	—
Uncached Input Tokens	73055	143421	+96.3%	—	—	—
Cost	0.4582	1.0884	+137.6%	—	—	—
Cached vs Uncached Token Ratio	11.50:1	18.98:1	+64.9%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc 'cat rename-plan.md && \
git status --short'
{
  "symbol": "LoadInput"
}
{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "CheckInput"
}
{
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "NormalizeValue"
}
{
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "RouteValue"
}
{
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "StampEvent"
}
{
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "CountItems"
}
{
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "MergeParts"
}
{
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "FilterNoise"
}
{
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "SortRecords"
}
{
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "DecodeReply"
}
{
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "OpenChannel"
}
{
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'rename-plan.md' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md' && \ git status --short"`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md && \ git status --short'`
2	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n 'Processor\|[A-Z][A-Za-z]+\\(' --glob '*.go' \| head -n 200"`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s2=s.replace(\"LoadInput\", \"ReadRequest\"); assert s2 "'!= s; fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=".workflow.go."); f=os.fdopen(fd,"w"); f.write(s2); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc "git status --short -- pipeline/workflow.go rename-plan.md go.mod && \ git diff -- pipeline/workflow.go && \ rg -n 'LoadInput\|ReadRequest\|CheckInput' pipeline/workflow.go"`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s2=s.replace(\"CheckInput\", \"ValidateRequest\"); assert s2 "'!= s; fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=".workflow.go."); f=os.fdopen(fd,"w"); f.write(s2); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s2=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); assert s2 "'!= s; fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=".workflow.go."); f=os.fdopen(fd,"w"); f.write(s2); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
7	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"RouteValue\", \"DispatchValue\"),(\"StampEvent\", \"AnnotateEvent\"),(\"CountItems\", \"TallyItems\"),(\"MergeParts\", \"CombineParts\")); s2=s; [None for old,new in pairs if old in s2]; assert all(old in s2 for old,new in pairs); [None for old,new in pairs if (s2:=s2.replace(old,new))]; fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.go.\"); f=os.fdopen(fd,\"w\"); f.write(s2); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)' && \ go test ./..."	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"FilterNoise\", \"PruneNoise\"),(\"SortRecords\", \"OrderRecords\"),(\"EncodePayload\", \"MarshalPayload\"),(\"DecodeReply\", \"ParseReply\"),(\"OpenChannel\", \"ConnectChannel\"),(\"CloseChannel\", \"ReleaseChannel\"),(\"FlushBuffer\", \"DrainBuffer\"),(\"SaveResult\", \"PersistResult\")); assert all(old in s for old,new in pairs); s2=s for old,new in pairs: s2=s2.replace(old,new) fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.go.\"); f=os.fdopen(fd,\"w\"); f.write(s2); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)' && \ go test ./..."	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
9	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
10	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
11	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MapFields", "to": "ProjectFields" }`
12	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
14	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
16	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CountItems", "to": "TallyItems" }`
19	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MergeParts", "to": "CombineParts" }`
21	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
25	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
27	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
29	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
32	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
34	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
36	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SaveResult", "to": "PersistResult" }`
38	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	59.14s	235.15s	+297.6%	—	—	—
Process Start → First Event	0.13s	0.13s	—	—	—	—
First Event → First Tool Call	4.99s	8.11s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	46	+557.1%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	46	+557.1%	—	—	—
Output Tokens	3912	5820	+48.8%	—	—	—
Reasoning / Thinking Tokens	175	649	+270.9%	—	—	—
Total Input Tokens	811786	2590943	+219.2%	—	—	—
Cached Input Tokens	735232	2453760	+233.7%	—	—	—
Uncached Input Tokens	76554	137183	+79.2%	—	—	—
Cost	0.4263	1.0373	+143.3%	—	—	—
Cached vs Uncached Token Ratio	9.60:1	17.89:1	+86.2%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc 'cat rename-plan.md'
{
  "symbol": "LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'
/bin/zsh -lc 'go test ./...'
/bin/zsh -lc "git status --short && \
rg -n 'ReadRequest|LoadInput' pipeline"
{
  "symbol": "CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'
/bin/zsh -lc 'go test ./...'
{
  "symbol": "NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'
/bin/zsh -lc 'go test ./...'
{
  "symbol": "RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "StampEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "CountItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'
/bin/zsh -lc 'go test ./...'
{
  "symbol": "FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|<row1 method name if present>\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md'`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ rg -n \"\\.LoadInput\\(\|LoadInput\\(\" --glob '*.go' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_inspect_symbol` (✓ ✓) `{ "file": "pipeline/workflow.go", "symbol": "Processor.LoadInput" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "symbol": "Processor.LoadInput" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"RouteValue\", \"DispatchValue\").replace(\"StampEvent\", \"AnnotateEvent\").replace(\"CountItems\", \"TallyItems\").replace(\"MergeParts\", \"CombineParts\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'`
7	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); renames=[(\"FilterNoise\", \"PruneNoise\"), (\"SortRecords\", \"OrderRecords\"), (\"EncodePayload\", \"MarshalPayload\"), (\"DecodeReply\", \"ParseReply\"), (\"OpenChannel\", \"ConnectChannel\"), (\"CloseChannel\", \"ReleaseChannel\"), (\"FlushBuffer\", \"DrainBuffer\"), (\"SaveResult\", \"PersistResult\")]; for old,new in renames: s=s.replace(old,new) p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	—	✓ ✓ `/bin/zsh -lc "git status --short && \ rg -n 'ReadRequest\|LoadInput' pipeline"`
9	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
10	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
11	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'`
12	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
14	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
16	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
17	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'`
18	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
19	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
21	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
22	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
24	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
26	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
27	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'`
28	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
35	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
37	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
39	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
40	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
41	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
42	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
43	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
44	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
45	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go go.mod rename-plan.md'`
46	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	66.72s	218.50s	+227.5%	—	—	—
Process Start → First Event	0.13s	0.12s	—	—	—	—
First Event → First Tool Call	4.40s	4.87s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	6	40	+566.7%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	6	40	+566.7%	—	—	—
Output Tokens	5279	4606	-12.7%	—	—	—
Reasoning / Thinking Tokens	425	480	+12.9%	—	—	—
Total Input Tokens	714768	1811888	+153.5%	—	—	—
Cached Input Tokens	649472	1697792	+161.4%	—	—	—
Uncached Input Tokens	65296	114096	+74.7%	—	—	—
Cost	0.3969	0.7733	+94.8%	—	—	—
Cached vs Uncached Token Ratio	9.95:1	14.88:1	+49.6%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' | head -80"
{
  "language": "go",
  "symbol": "(*Processor).LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
/bin/zsh -lc 'git status --short && \
git diff -- pipeline/workflow.go && \
git diff --name-only'
/bin/zsh -lc "rg -n 'LoadInput|ReadRequest' pipeline"
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|\" --glob '*.go'"`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' \| head -80"`
2	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path import os, tempfile, time p = Path('pipeline/workflow.go') s = p.read_text() s2 = s.replace('func (Processor) LoadInput(', 'func (Processor) ReadRequest(').replace('processor.LoadInput(', 'processor.ReadRequest(') if s2 == s or s.count('LoadInput') "'!= 2: raise SystemExit('"'unexpected LoadInput occurrence count') fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.') try: with os.fdopen(fd, 'w') as f: f.write(s2) f.flush() os.fsync(f.fileno()) os.replace(tmp, p) os.utime(p, ns=(time.time_ns(), time.time_ns())) finally: if os.path.exists(tmp): os.unlink(tmp) PY go test ./..."	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "(*Processor).LoadInput" }`
3	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path import os, tempfile, time p = Path('pipeline/workflow.go') s = p.read_text() if s.count('CheckInput') "'!= 2: raise SystemExit('"'unexpected CheckInput occurrence count') s2 = s.replace('CheckInput', 'ValidateRequest') fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.') try: with os.fdopen(fd, 'w') as f: f.write(s2) f.flush() os.fsync(f.fileno()) os.replace(tmp, p) now = time.time_ns() os.utime(p, ns=(now, now)) finally: if os.path.exists(tmp): os.unlink(tmp) PY go test ./..."	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path import os, tempfile, time p = Path('pipeline/workflow.go') s = p.read_text() for old in ('NormalizeValue', 'MapFields'): if s.count(old) "'!= 2: raise SystemExit(f'"'unexpected {old} occurrence count') s2 = s.replace('NormalizeValue', 'CanonicalizeValue').replace('MapFields', 'ProjectFields') fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.') try: with os.fdopen(fd, 'w') as f: f.write(s2) f.flush() os.fsync(f.fileno()) os.replace(tmp, p) now = time.time_ns() os.utime(p, ns=(now, now)) finally: if os.path.exists(tmp): os.unlink(tmp) PY go test ./..."	✓ ✓ `/bin/zsh -lc 'go test ./...'`
5	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path import os, tempfile, time p = Path('pipeline/workflow.go') s = p.read_text() renames = { 'RouteValue': 'DispatchValue', 'StampEvent': 'AnnotateEvent', 'CountItems': 'TallyItems', 'MergeParts': 'CombineParts', } for old in renames: if s.count(old) "'!= 2: raise SystemExit(f'"'unexpected {old} occurrence count') s2 = s for old, new in renames.items(): s2 = s2.replace(old, new) fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.') try: with os.fdopen(fd, 'w') as f: f.write(s2) f.flush() os.fsync(f.fileno()) os.replace(tmp, p) now = time.time_ns() os.utime(p, ns=(now, now)) finally: if os.path.exists(tmp): os.unlink(tmp) PY go test ./..."	✓ ✓ `/bin/zsh -lc 'git status --short && \ git diff -- pipeline/workflow.go && \ git diff --name-only'`
6	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path import os, tempfile, time p = Path('pipeline/workflow.go') s = p.read_text() renames = { 'FilterNoise': 'PruneNoise', 'SortRecords': 'OrderRecords', 'EncodePayload': 'MarshalPayload', 'DecodeReply': 'ParseReply', 'OpenChannel': 'ConnectChannel', 'CloseChannel': 'ReleaseChannel', 'FlushBuffer': 'DrainBuffer', 'SaveResult': 'PersistResult', } for old in renames: if s.count(old) "'!= 2: raise SystemExit(f'"'unexpected {old} occurrence count') s2 = s for old, new in renames.items(): s2 = s2.replace(old, new) fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.') try: with os.fdopen(fd, 'w') as f: f.write(s2) f.flush() os.fsync(f.fileno()) os.replace(tmp, p) now = time.time_ns() os.utime(p, ns=(now, now)) finally: if os.path.exists(tmp): os.unlink(tmp) PY go test ./..."	✓ ✓ `/bin/zsh -lc "rg -n 'LoadInput\|ReadRequest' pipeline"`
7	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
8	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
9	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
10	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
11	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
12	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
13	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
14	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
22	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
23	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
39	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
40	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	75.73s	228.97s	+202.3%	—	—	—
Process Start → First Event	0.13s	0.13s	—	—	—	—
First Event → First Tool Call	5.61s	5.67s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	11	39	+254.5%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	11	39	+254.5%	—	—	—
Output Tokens	6326	5936	-6.2%	—	—	—
Reasoning / Thinking Tokens	390	1478	+279.0%	—	—	—
Total Input Tokens	1040832	2137953	+105.4%	—	—	—
Cached Input Tokens	929280	1991680	+114.3%	—	—	—
Uncached Input Tokens	111552	146273	+31.1%	—	—	—
Cost	0.5951	0.9563	+60.7%	—	—	—
Cached vs Uncached Token Ratio	8.33:1	13.62:1	+63.5%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/benchmark*' -g 'tools/benchmark-harness/README.md' -g 'go.mod'"
/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md'"
{
  "language": "go",
  "symbol": "LoadInput"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "CheckInput"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "NormalizeValue"
}
{
  "language": "go",
  "symbol": "MapFields"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "StampEvent"
}
{
  "language": "go",
  "symbol": "RouteValue"
}
{
  "language": "go",
  "symbol": "CountItems"
}
{
  "language": "go",
  "symbol": "MergeParts"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "FilterNoise"
}
{
  "language": "go",
  "symbol": "SortRecords"
}
{
  "language": "go",
  "symbol": "EncodePayload"
}
{
  "language": "go",
  "symbol": "DecodeReply"
}
{
  "language": "go",
  "symbol": "OpenChannel"
}
{
  "language": "go",
  "symbol": "CloseChannel"
}
{
  "language": "go",
  "symbol": "FlushBuffer"
}
{
  "language": "go",
  "symbol": "SaveResult"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "auto_organize_imports": false,
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|method-name-from-plan\" . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/benchmark*' -g 'tools/benchmark-harness/README.md' -g 'go.mod'"`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ cat pipeline/workflow_test.go && \ rg -n '\\.LoadInput\\(\|LoadInput\\(' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"' --glob '"'!*/_test.go'"'"`	? ✗ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md'"`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); updated=data.replace(\"LoadInput\", \"ReadRequest\"); assert updated "'!= data and data.count("LoadInput") == 2; fd, tmp=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)'"' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc "rg -n 'CheckInput' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"`	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); updated=data.replace(\"CheckInput\", \"ValidateRequest\"); assert data.count(\"CheckInput\") == 2; fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
6	✓ ✓ `/bin/zsh -lc "rg -n 'NormalizeValue\|MapFields' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CheckInput" }`
7	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); assert data.count(\"NormalizeValue\") == 2 and data.count(\"MapFields\") == 2; updated=data.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
8	✓ ✓ `/bin/zsh -lc "rg -n 'RouteValue\|StampEvent\|CountItems\|MergeParts' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
9	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); names={\"RouteValue\":\"DispatchValue\",\"StampEvent\":\"AnnotateEvent\",\"CountItems\":\"TallyItems\",\"MergeParts\":\"CombineParts\"}; assert all(data.count(old)==2 for old in names); updated=data for old,new in names.items(): updated=updated.replace(old,new) fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "NormalizeValue" }`
10	✓ ✓ `/bin/zsh -lc "rg -n 'FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MapFields" }`
11	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); names={\"FilterNoise\":\"PruneNoise\",\"SortRecords\":\"OrderRecords\",\"EncodePayload\":\"MarshalPayload\",\"DecodeReply\":\"ParseReply\",\"OpenChannel\":\"ConnectChannel\",\"CloseChannel\":\"ReleaseChannel\",\"FlushBuffer\":\"DrainBuffer\",\"SaveResult\":\"PersistResult\"}; assert all(data.count(old)==2 for old in names); updated=data for old,new in names.items(): updated=updated.replace(old,new) fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
12	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
13	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "StampEvent" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "RouteValue" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CountItems" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MergeParts" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
22	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FilterNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SortRecords" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "EncodePayload" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "DecodeReply" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "OpenChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CloseChannel" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FlushBuffer" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SaveResult" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "auto_organize_imports": false, "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
39	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Benchmark run luna-medium-five-rename-multifile-r10-20260929T1134Z Benchmark run luna-medium-staged-followup-smoke-20260929

Benchmark run luna-medium-staged-followup-r10-20260929T054733Z

Interactive browser

Baseline vs semedit

Detailed observations

Benchmark Methodology & Transparency

Test case: task-13-rename-followup-scale

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Test case: `task-13-rename-followup-scale`

Target: `codex/gpt-6-luna/medium`