Benchmark run luna-medium-staged-prewarm-r10-20260929

This page contains exactly the publishable benchmark observations recorded in run luna-medium-staged-prewarm-r10-20260929; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
4. Level 4 (Test Suite): Unit and integration test suite execution.
Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: `task-13-rename-followup-scale`

Target: `codex/gpt-6-luna/medium`

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	108.13s	229.42s	+112.2%	—	—	—
Process Start → First Event	0.83s	0.14s	—	—	—	—
First Event → First Tool Call	10.24s	5.10s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	16	41	+156.2%	—	—	—
Initial Load / Discovery Turns	2	0	-100.0%	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	16	41	+156.2%	—	—	—
Output Tokens	5132	8159	+59.0%	—	—	—
Reasoning / Thinking Tokens	72	2949	+3995.8%	—	—	—
Total Input Tokens	1427470	2314395	+62.1%	—	—	—
Cached Input Tokens	1338112	2174976	+62.5%	—	—	—
Uncached Input Tokens	89358	139419	+56.0%	—	—	—
Cost	0.6230	1.0311	+65.5%	—	—	—
Cached vs Uncached Token Ratio	14.97:1	15.60:1	+4.2%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'go.mod' -g '*.go' | head -80"
/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md'"
{
  "symbol": "LoadInput"
}
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git status --short && \
git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'
{
  "file": "pipeline/workflow.go",
  "symbol": "Processor.ReadRequest"
}
/bin/zsh -lc "git status --short --untracked-files=all -- pipeline/workflow.go && \
git diff --no-ext-diff -- pipeline/workflow.go && \
rg -n 'LoadInput|ReadRequest' pipeline"
{
  "symbol": "CheckInput"
}
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc "rg -n 'CheckInput|ValidateRequest|LoadInput|ReadRequest' pipeline && \
git diff -- go.mod rename-plan.md && \
go test ./..."
{
  "symbol": "NormalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc "rg -n 'NormalizeValue|CanonicalizeValue|MapFields|ProjectFields|RouteValue|StampEvent' pipeline && \
git diff -- go.mod rename-plan.md && \
go test ./..."
{
  "symbol": "RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "CountItems"
}
{
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "MergeParts"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc "rg -n 'RouteValue|DispatchValue|StampEvent|AnnotateEvent|CountItems|TallyItems|MergeParts|CombineParts|FilterNoise|SortRecords' pipeline && \
git diff -- go.mod rename-plan.md && \
go test ./..."
{
  "symbol": "FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc "rg -n 'FilterNoise|PruneNoise|SortRecords|OrderRecords|EncodePayload|MarshalPayload|DecodeReply|ParseReply|OpenChannel|ConnectChannel|CloseChannel|ReleaseChannel|FlushBuffer|DrainBuffer|SaveResult|PersistResult' pipeline && \
git diff -- go.mod rename-plan.md && \
go test ./..."

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|row 1\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'go.mod' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go && \ rg -n "LoadInput" .'`	? ✗ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md'"`
3	`file_change` (✓ ✓)	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "rg -n \"CheckInput\|ValidateRequest\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc 'git status --short && \ git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`
6	`file_change` (✓ ✓)	`semedit/semantic_inspect_symbol` (✓ ✓) `{ "file": "pipeline/workflow.go", "symbol": "Processor.ReadRequest" }`
7	✓ ✓ `/bin/zsh -lc 'go test ./...'`	✓ ✓ `/bin/zsh -lc "git status --short --untracked-files=all -- pipeline/workflow.go && \ git diff --no-ext-diff -- pipeline/workflow.go && \ rg -n 'LoadInput\|ReadRequest' pipeline"`
8	✓ ✓ `/bin/zsh -lc "rg -n \"NormalizeValue\|MapFields\|CanonicalizeValue\|ProjectFields\" --glob '*.go' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
9	`file_change` (✓ ✓)	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
10	✓ ✓ `/bin/zsh -lc 'go test ./...'`	✓ ✓ `/bin/zsh -lc "rg -n 'CheckInput\|ValidateRequest\|LoadInput\|ReadRequest' pipeline && \ git diff -- go.mod rename-plan.md && \ go test ./..."`
11	✓ ✓ `/bin/zsh -lc "rg -n \"RouteValue\|StampEvent\|CountItems\|MergeParts\|DispatchValue\|AnnotateEvent\|TallyItems\|CombineParts\" --glob '*.go' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
12	`file_change` (✓ ✓)	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
13	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
14	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult\|PruneNoise\|OrderRecords\|MarshalPayload\|ParseReply\|ConnectChannel\|ReleaseChannel\|DrainBuffer\|PersistResult\" --glob '*.go' ."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
15	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "rg -n 'NormalizeValue\|CanonicalizeValue\|MapFields\|ProjectFields\|RouteValue\|StampEvent' pipeline && \ git diff -- go.mod rename-plan.md && \ go test ./..."`
16	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
20	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
24	—	✓ ✓ `/bin/zsh -lc "rg -n 'RouteValue\|DispatchValue\|StampEvent\|AnnotateEvent\|CountItems\|TallyItems\|MergeParts\|CombineParts\|FilterNoise\|SortRecords' pipeline && \ git diff -- go.mod rename-plan.md && \ go test ./..."`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
26	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
28	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
31	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
35	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
37	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
39	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
40	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
41	—	✓ ✓ `/bin/zsh -lc "rg -n 'FilterNoise\|PruneNoise\|SortRecords\|OrderRecords\|EncodePayload\|MarshalPayload\|DecodeReply\|ParseReply\|OpenChannel\|ConnectChannel\|CloseChannel\|ReleaseChannel\|FlushBuffer\|DrainBuffer\|SaveResult\|PersistResult' pipeline && \ git diff -- go.mod rename-plan.md && \ go test ./..."`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	60.32s	272.15s	+351.2%	—	—	—
Process Start → First Event	0.83s	0.14s	—	—	—	—
First Event → First Tool Call	6.00s	4.79s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	40	+471.4%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	40	+471.4%	—	—	—
Output Tokens	3897	6700	+71.9%	—	—	—
Reasoning / Thinking Tokens	0	770	—	—	—	—
Total Input Tokens	857816	3615888	+321.5%	—	—	—
Cached Input Tokens	781312	3439360	+340.2%	—	—	—
Uncached Input Tokens	76504	176528	+130.7%	—	—	—
Cost	0.4353	1.3945	+220.4%	—	—	—
Cached vs Uncached Token Ratio	10.21:1	19.48:1	+90.8%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -80"
/bin/zsh -lc 'cat rename-plan.md && \
git status --short && \
cat AGENTS.override.md'
{
  "symbol": "LoadInput"
}
{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'
/bin/zsh -lc "sed -n '1,120p' pipeline/workflow.go && \
git status --short -- pipeline/workflow.go rename-plan.md go.mod"
{
  "symbol": "CheckInput"
}
{
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'
{
  "symbol": "NormalizeValue"
}
{
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'
{
  "symbol": "RouteValue"
}
{
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "StampEvent"
}
{
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "CountItems"
}
{
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "MergeParts"
}
{
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'
{
  "symbol": "FilterNoise"
}
{
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "SortRecords"
}
{
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "DecodeReply"
}
{
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "OpenChannel"
}
{
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \
go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|method\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go && \ cat pipeline/workflow_test.go && \ rg -n "LoadInput" .'`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md && \ git status --short && \ cat AGENTS.override.md'`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text().replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); names={\"RouteValue\":\"DispatchValue\",\"StampEvent\":\"AnnotateEvent\",\"CountItems\":\"TallyItems\",\"MergeParts\":\"CombineParts\"}; for old,new in names.items(): s=s.replace(old,new) t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc "sed -n '1,120p' pipeline/workflow.go && \ git status --short -- pipeline/workflow.go rename-plan.md go.mod"`
7	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); names={\"FilterNoise\":\"PruneNoise\",\"SortRecords\":\"OrderRecords\",\"EncodePayload\":\"MarshalPayload\",\"DecodeReply\":\"ParseReply\",\"OpenChannel\":\"ConnectChannel\",\"CloseChannel\":\"ReleaseChannel\",\"FlushBuffer\":\"DrainBuffer\",\"SaveResult\":\"PersistResult\"}; for old,new in names.items(): s=s.replace(old,new) t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \ go test ./..."	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
8	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
9	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`
10	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
11	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
12	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
13	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MapFields", "to": "ProjectFields" }`
14	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
16	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
19	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CountItems", "to": "TallyItems" }`
21	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
22	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MergeParts", "to": "CombineParts" }`
23	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
25	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
27	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
29	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
32	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
34	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
36	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
38	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
39	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SaveResult", "to": "PersistResult" }`
40	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go rename-plan.md go.mod && \ go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	106.41s	279.25s	+162.4%	—	—	—
Process Start → First Event	0.83s	0.13s	—	—	—	—
First Event → First Tool Call	6.00s	4.75s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	18	51	+183.3%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	18	51	+183.3%	—	—	—
Output Tokens	8950	6006	-32.9%	—	—	—
Reasoning / Thinking Tokens	1222	950	-22.3%	—	—	—
Total Input Tokens	1723653	2918419	+69.3%	—	—	—
Cached Input Tokens	1613056	2734592	+69.5%	—	—	—
Uncached Input Tokens	110597	183827	+66.2%	—	—	—
Cost	0.8069	1.2302	+52.5%	—	—	—
Cached vs Uncached Token Ratio	14.58:1	14.88:1	+2.0%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg -n \"\" rename-plan.md && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' | head -80"
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \
go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \
go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g '*.go'"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg -n \"\" rename-plan.md && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "rg -n '\\bLoadInput\\b\|\\b(ReadRequest)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"LoadInput\", \"ReadRequest\"); assert s "'!= old and old.count("LoadInput") == 2; p.write_text(s)'"'"`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput" }`
4	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go && \ git status --short'`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \ go test ./...'`
6	✓ ✓ `/bin/zsh -lc "sed -n '1,35p' pipeline/workflow.go && \ git status --short --untracked-files=all -- pipeline/workflow.go pipeline/workflow_test.go go.mod rename-plan.md && \ rg -n '\\bLoadInput\\b\|\\bReadRequest\\b' pipeline"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
7	✓ ✓ `/bin/zsh -lc "rg -n '\\bCheckInput\\b\|\\bValidateRequest\\b' . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"' && \ sed -n '1,50p' pipeline/workflow.go"`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput" }`
8	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"CheckInput\") == 2; p.write_text(s.replace(\"CheckInput\", \"ValidateRequest\"))'"`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
9	✓ ✓ `/bin/zsh -lc 'go test ./...'`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \ go test ./...'`
10	✓ ✓ `/bin/zsh -lc "rg -n '\\b(NormalizeValue\|CanonicalizeValue\|MapFields\|ProjectFields)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
11	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"NormalizeValue\") == 2 and s.count(\"MapFields\") == 2; s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)'"`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue" }`
12	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
13	✓ ✓ `/bin/zsh -lc "rg -n '\\b(RouteValue\|DispatchValue\|StampEvent\|AnnotateEvent\|CountItems\|TallyItems\|MergeParts\|CombineParts)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
14	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"RouteValue\",\"DispatchValue\"),(\"StampEvent\",\"AnnotateEvent\"),(\"CountItems\",\"TallyItems\"),(\"MergeParts\",\"CombineParts\")); assert all(s.count(a)==2 for a,b in pairs); [None for a,b in pairs]; for a,b in pairs: s=s.replace(a,b) p.write_text(s)'"`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields" }`
15	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
16	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n '\\b(FilterNoise\|PruneNoise\|SortRecords\|OrderRecords\|EncodePayload\|MarshalPayload\|DecodeReply\|ParseReply\|OpenChannel\|ConnectChannel\|CloseChannel\|ReleaseChannel\|FlushBuffer\|DrainBuffer\|SaveResult\|PersistResult)\\b' . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \ go test ./...'`
17	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"FilterNoise\",\"PruneNoise\"),(\"SortRecords\",\"OrderRecords\"),(\"EncodePayload\",\"MarshalPayload\"),(\"DecodeReply\",\"ParseReply\"),(\"OpenChannel\",\"ConnectChannel\"),(\"CloseChannel\",\"ReleaseChannel\"),(\"FlushBuffer\",\"DrainBuffer\"),(\"SaveResult\",\"PersistResult\")); assert all(s.count(a)==2 for a,b in pairs); for a,b in pairs: s=s.replace(a,b) p.write_text(s)'"`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
18	✓ ✓ `/bin/zsh -lc 'go test ./...'`	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
20	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
22	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems" }`
23	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
24	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts" }`
25	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
26	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \ go test ./...'`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
28	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise" }`
29	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
31	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
33	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
34	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
36	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
37	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
39	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
40	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel" }`
41	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
42	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
43	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel" }`
44	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
45	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
46	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer" }`
47	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
48	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
49	—	`semedit/semantic_find_references` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult" }`
50	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
51	—	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md && \ go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	66.90s	226.08s	+237.9%	—	—	—
Process Start → First Event	0.13s	0.14s	—	—	—	—
First Event → First Tool Call	7.19s	5.62s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	13	38	+192.3%	—	—	—
Initial Load / Discovery Turns	3	0	-100.0%	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	13	38	+192.3%	—	—	—
Output Tokens	4138	3551	-14.2%	—	—	—
Reasoning / Thinking Tokens	275	0	-100.0%	—	—	—
Total Input Tokens	1214286	1722184	+41.8%	—	—	—
Cached Input Tokens	1133568	1609984	+42.0%	—	—	—
Uncached Input Tokens	80718	112200	+39.0%	—	—	—
Cost	0.5403	0.7274	+34.6%	—	—	—
Cached vs Uncached Token Ratio	14.04:1	14.35:1	+2.2%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc 'cat rename-plan.md'
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g '.go' \| head -80"`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md'`
2	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md && \ cat AGENTS.override.md && \ cat pipeline/workflow.go'`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "rg -n '\\bLoadInput\\b' ."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); p.write_text(s)' && \ git diff -- pipeline/workflow.go rename-plan.md go.mod"`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
5	✓ ✓ `/bin/zsh -lc "rg -n '\\b(ReadRequest\|LoadInput)\\b' pipeline/workflow.go && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
6	✓ ✓ `/bin/zsh -lc "rg -n '\\b(CheckInput\|ValidateRequest)\\b' ."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
7	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	✓ ✓ `/bin/zsh -lc "rg -n '\\b(NormalizeValue\|MapFields\|CanonicalizeValue\|ProjectFields)\\b' ."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
9	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
10	✓ ✓ `/bin/zsh -lc "rg -n '\\b(RouteValue\|StampEvent\|CountItems\|MergeParts\|DispatchValue\|AnnotateEvent\|TallyItems\|CombineParts)\\b' ."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
11	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"RouteValue\", \"DispatchValue\"), (\"StampEvent\", \"AnnotateEvent\"), (\"CountItems\", \"TallyItems\"), (\"MergeParts\", \"CombineParts\")); [None for old, new in pairs if not s.__contains__(old)]; for old, new in pairs: s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
12	✓ ✓ `/bin/zsh -lc "rg -n '\\b(FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult\|PruneNoise\|OrderRecords\|MarshalPayload\|ParseReply\|ConnectChannel\|ReleaseChannel\|DrainBuffer\|PersistResult)\\b' ."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=((\"FilterNoise\", \"PruneNoise\"), (\"SortRecords\", \"OrderRecords\"), (\"EncodePayload\", \"MarshalPayload\"), (\"DecodeReply\", \"ParseReply\"), (\"OpenChannel\", \"ConnectChannel\"), (\"CloseChannel\", \"ReleaseChannel\"), (\"FlushBuffer\", \"DrainBuffer\"), (\"SaveResult\", \"PersistResult\")); for old, new in pairs: s=s.replace(old, new) p.write_text(s)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
21	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
38	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	80.31s	257.00s	+220.0%	—	—	—
Process Start → First Event	0.14s	0.13s	—	—	—	—
First Event → First Tool Call	4.37s	9.96s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	12	39	+225.0%	—	—	—
Initial Load / Discovery Turns	3	0	-100.0%	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	12	39	+225.0%	—	—	—
Output Tokens	6188	5486	-11.3%	—	—	—
Reasoning / Thinking Tokens	844	0	-100.0%	—	—	—
Total Input Tokens	1155316	2364213	+104.6%	—	—	—
Cached Input Tokens	1034496	2194432	+112.1%	—	—	—
Uncached Input Tokens	120820	169781	+40.5%	—	—	—
Cost	0.6486	1.0416	+60.6%	—	—	—
Cached vs Uncached Token Ratio	8.56:1	12.93:1	+51.0%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'go.mod' -g '*.go' | head -100"
/bin/zsh -lc 'cat rename-plan.md'
{
  "symbol": "LoadInput"
}
{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "CheckInput"
}
{
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "NormalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "RouteValue"
}
{
  "symbol": "StampEvent"
}
{
  "symbol": "CountItems"
}
{
  "symbol": "MergeParts"
}
{
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "FilterNoise"
}
{
  "symbol": "SortRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "symbol": "DecodeReply"
}
{
  "symbol": "OpenChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ git status --short && \ cat rename-plan.md && \ rg -n \"Processor\|method\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'go.mod' -g '*.go' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ rg -n \"\\.LoadInput\\(\|LoadInput\\(\" --glob '"'!rename-plan.md'"' ."`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md'`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"LoadInput\", \"ReadRequest\"); assert s "'!= old and old.count("LoadInput") == 2; fd, name=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc 'git diff -- pipeline/workflow.go && \ git status --short -- pipeline/workflow.go rename-plan.md go.mod'`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "rg -n \"LoadInput\|ReadRequest\|CheckInput\|ValidateRequest\" pipeline --glob '*.go' && \ git status --short -- pipeline/workflow.go rename-plan.md go.mod"`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"CheckInput\", \"ValidateRequest\"); assert s "'!= old and old.count("CheckInput") == 2; fd, name=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
7	✓ ✓ `/bin/zsh -lc "rg -n \"NormalizeValue\|CanonicalizeValue\|MapFields\|ProjectFields\" pipeline --glob '*.go'"`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
8	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); assert s "'!= old and old.count("NormalizeValue") == 2 and old.count("MapFields") == 2; fd, name=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
9	✓ ✓ `/bin/zsh -lc "rg -n \"RouteValue\|StampEvent\|CountItems\|MergeParts\" pipeline --glob '*.go'"`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
10	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; pairs=((\"RouteValue\",\"DispatchValue\"),(\"StampEvent\",\"AnnotateEvent\"),(\"CountItems\",\"TallyItems\"),(\"MergeParts\",\"CombineParts\")); counts={a:old.count(a) for a,b in pairs}; assert all(n == 2 for n in counts.values()); for a,b in pairs: s=s.replace(a,b) assert s "'!= old; fd, name=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
11	✓ ✓ `/bin/zsh -lc "rg -n \"FilterNoise\|SortRecords\|EncodePayload\|DecodeReply\|OpenChannel\|CloseChannel\|FlushBuffer\|SaveResult\" pipeline --glob '*.go'"`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
12	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; pairs=((\"FilterNoise\",\"PruneNoise\"),(\"SortRecords\",\"OrderRecords\"),(\"EncodePayload\",\"MarshalPayload\"),(\"DecodeReply\",\"ParseReply\"),(\"OpenChannel\",\"ConnectChannel\"),(\"CloseChannel\",\"ReleaseChannel\"),(\"FlushBuffer\",\"DrainBuffer\"),(\"SaveResult\",\"PersistResult\")); assert all(old.count(a) == 2 for a,b in pairs); for a,b in pairs: s=s.replace(a,b) assert s "'!= old; fd, name=tempfile.mkstemp(dir=p.parent, prefix=".workflow."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"' && \ go test ./..."	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MapFields", "to": "ProjectFields" }`
13	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
17	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CountItems", "to": "TallyItems" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MergeParts", "to": "CombineParts" }`
22	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
38	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SaveResult", "to": "PersistResult" }`
39	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	53.84s	230.48s	+328.0%	—	—	—
Process Start → First Event	0.12s	0.13s	—	—	—	—
First Event → First Tool Call	3.97s	5.37s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	40	+471.4%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	40	+471.4%	—	—	—
Output Tokens	3518	5341	+51.8%	—	—	—
Reasoning / Thinking Tokens	240	231	-3.8%	—	—	—
Total Input Tokens	811665	2739774	+237.5%	—	—	—
Cached Input Tokens	748544	2586112	+245.5%	—	—	—
Uncached Input Tokens	63121	153662	+143.4%	—	—	—
Cost	0.3919	1.1003	+180.8%	—	—	—
Cached vs Uncached Token Ratio	11.86:1	16.83:1	+41.9%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md'"
/bin/zsh -lc "rg -n 'LoadInput|ReadRequest' --glob '*.go' --glob '"'!**/*_test.go'"'"
{
  "language": "go",
  "symbol": "LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "RouteValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "language": "go",
  "symbol": "StampEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "language": "go",
  "symbol": "CountItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "language": "go",
  "symbol": "MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "FilterNoise"
}
{
  "language": "go",
  "symbol": "FilterNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "language": "go",
  "symbol": "SortRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "language": "go",
  "symbol": "EncodePayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "language": "go",
  "symbol": "DecodeReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "language": "go",
  "symbol": "OpenChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "language": "go",
  "symbol": "CloseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "language": "go",
  "symbol": "FlushBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "language": "go",
  "symbol": "SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|method\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md'"`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go && \ cat pipeline/workflow_test.go && \ rg -n "LoadInput" .'`	✓ ✓ `/bin/zsh -lc "rg -n 'LoadInput\|ReadRequest' --glob '.go' --glob '"'!/_test.go'"'"`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "LoadInput" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"CheckInput\", \"ValidateRequest\"); fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text().replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
6	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text().replace(\"RouteValue\", \"DispatchValue\").replace(\"StampEvent\", \"AnnotateEvent\").replace(\"CountItems\", \"TallyItems\").replace(\"MergeParts\", \"CombineParts\"); fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CheckInput" }`
7	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=[(\"FilterNoise\",\"PruneNoise\"),(\"SortRecords\",\"OrderRecords\"),(\"EncodePayload\",\"MarshalPayload\"),(\"DecodeReply\",\"ParseReply\"),(\"OpenChannel\",\"ConnectChannel\"),(\"CloseChannel\",\"ReleaseChannel\"),(\"FlushBuffer\",\"DrainBuffer\"),(\"SaveResult\",\"PersistResult\")]; for old,new in pairs: s=s.replace(old,new) fd,tmp=tempfile.mkstemp(dir=str(p.parent),prefix=\".workflow.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \ go test ./..."	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
8	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
9	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "NormalizeValue" }`
10	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
11	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MapFields" }`
12	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
13	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "RouteValue" }`
15	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "StampEvent" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
18	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CountItems" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
20	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "MergeParts" }`
21	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
22	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FilterNoise" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FilterNoise" }`
25	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SortRecords" }`
27	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "EncodePayload" }`
29	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
30	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "DecodeReply" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
32	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "OpenChannel" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
34	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "CloseChannel" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
36	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "FlushBuffer" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
38	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "SaveResult" }`
39	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
40	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	51.66s	264.83s	+412.6%	—	—	—
Process Start → First Event	0.13s	0.14s	—	—	—	—
First Event → First Tool Call	4.60s	6.87s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	38	+442.9%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	38	+442.9%	—	—	—
Output Tokens	3195	3459	+8.3%	—	—	—
Reasoning / Thinking Tokens	0	0	—	—	—	—
Total Input Tokens	807280	2101890	+160.4%	—	—	—
Cached Input Tokens	739328	1969664	+166.4%	—	—	—
Uncached Input Tokens	67952	132226	+94.6%	—	—	—
Cost	0.3946	0.8662	+119.5%	—	—	—
Cached vs Uncached Token Ratio	10.88:1	14.90:1	+36.9%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' | head -80"
{
  "symbol": "LoadInput"
}
{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "CheckInput"
}
{
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "NormalizeValue"
}
{
  "symbol": "MapFields"
}
{
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "RouteValue"
}
{
  "symbol": "CountItems"
}
{
  "symbol": "StampEvent"
}
{
  "symbol": "MergeParts"
}
{
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "symbol": "FilterNoise"
}
{
  "symbol": "SortRecords"
}
{
  "symbol": "EncodePayload"
}
{
  "symbol": "DecodeReply"
}
{
  "symbol": "OpenChannel"
}
{
  "symbol": "CloseChannel"
}
{
  "symbol": "FlushBuffer"
}
{
  "symbol": "SaveResult"
}
{
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg -n \"Processor\|<method-name-from-row-1>\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc "cat rename-plan.md && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "cat pipeline/workflow.go && \ rg -n \"LoadInput\" . --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"LoadInput\", \"ReadRequest\"); assert s "'!= old and s.count("ReadRequest") == 2; tmp=p.with_suffix(".go.tmp"); tmp.write_text(s); tmp.replace(p)'"' && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"CheckInput\") == 2; s=s.replace(\"CheckInput\", \"ValidateRequest\"); tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s); tmp.replace(p)' && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
5	✓ ✓ `/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); assert s.count(\"NormalizeValue\") == 2 and s.count(\"MapFields\") == 2; s=s.replace(\"NormalizeValue\", \"CanonicalizeValue\").replace(\"MapFields\", \"ProjectFields\"); tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s); tmp.replace(p)' && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CheckInput" }`
6	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=[(\"RouteValue\",\"DispatchValue\"),(\"StampEvent\",\"AnnotateEvent\"),(\"CountItems\",\"TallyItems\"),(\"MergeParts\",\"CombineParts\")]; assert all(s.count(a)==2 for a,b in pairs); [None for a,b in pairs]; s=s.replace(\"RouteValue\", \"DispatchValue\").replace(\"StampEvent\", \"AnnotateEvent\").replace(\"CountItems\", \"TallyItems\").replace(\"MergeParts\", \"CombineParts\"); tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s); tmp.replace(p)' && \ go test ./..."	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
7	✓ ✓ /bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); pairs=[(\"FilterNoise\",\"PruneNoise\"),(\"SortRecords\",\"OrderRecords\"),(\"EncodePayload\",\"MarshalPayload\"),(\"DecodeReply\",\"ParseReply\"),(\"OpenChannel\",\"ConnectChannel\"),(\"CloseChannel\",\"ReleaseChannel\"),(\"FlushBuffer\",\"DrainBuffer\"),(\"SaveResult\",\"PersistResult\")]; assert all(s.count(a)==2 for a,b in pairs); s=s.replace(\"FilterNoise\", \"PruneNoise\").replace(\"SortRecords\", \"OrderRecords\").replace(\"EncodePayload\", \"MarshalPayload\").replace(\"DecodeReply\", \"ParseReply\").replace(\"OpenChannel\", \"ConnectChannel\").replace(\"CloseChannel\", \"ReleaseChannel\").replace(\"FlushBuffer\", \"DrainBuffer\").replace(\"SaveResult\", \"PersistResult\"); tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s); tmp.replace(p)' && \ go test ./..."	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "NormalizeValue" }`
9	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MapFields" }`
10	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
11	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MapFields", "to": "ProjectFields" }`
12	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "RouteValue" }`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CountItems" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "StampEvent" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "MergeParts" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CountItems", "to": "TallyItems" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.MergeParts", "to": "CombineParts" }`
21	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FilterNoise" }`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SortRecords" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "EncodePayload" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "DecodeReply" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "OpenChannel" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "CloseChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "FlushBuffer" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "symbol": "SaveResult" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "symbol": "Processor.SaveResult", "to": "PersistResult" }`
38	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · prescriptive MCP instructions · write semedit restriction

Setting	Value
Test case	`task-13-rename-followup-scale`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`prescriptive`
Semedit arm restriction	`write`
Fixture	testdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Semedit MCP Prompt:

Initial Workspace State (Before Edit)

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}

Metric	Vanilla (Small)	MCP (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	54.88s	239.36s	+336.2%	—	—	—
Process Start → First Event	0.14s	0.14s	—	—	—	—
First Event → First Tool Call	4.61s	14.43s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	5	5	0%	—	—	—
Internal Tool Cycles	7	38	+442.9%	—	—	—
Initial Load / Discovery Turns	0	0	—	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	7	38	+442.9%	—	—	—
Output Tokens	3530	4619	+30.8%	—	—	—
Reasoning / Thinking Tokens	0	673	—	—	—	—
Total Input Tokens	809064	2443665	+202.0%	—	—	—
Cached Input Tokens	683008	2242048	+228.3%	—	—	—
Uncached Input Tokens	126056	201617	+59.9%	—	—	—
Cost	0.5300	1.1307	+113.3%	—	—	—
Cached vs Uncached Token Ratio	5.42:1	11.12:1	+105.2%	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	PASS	PASS	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	PASS	—	—	—	—

Baseline vs Semedit Agentic in Small Context

Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)


/bin/zsh -lc 'cat rename-plan.md && \
git status --short'
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
/bin/zsh -lc 'go test ./...'
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
/bin/zsh -lc 'go test ./...'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ cat rename-plan.md && \ rg -n \"Processor\|method\" --glob '*.go' ."`	✓ ✓ `/bin/zsh -lc 'cat rename-plan.md && \ git status --short'`
2	✓ ✓ `/bin/zsh -lc 'cat pipeline/workflow.go'`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.LoadInput" }`
3	✓ ✓ `/bin/zsh -lc "python3 - <<'PY' from pathlib import Path p = Path('pipeline/workflow.go') s = p.read_text() s = s.replace('func (Processor) LoadInput(', 'func (Processor) ReadRequest(') s = s.replace('processor.LoadInput(', 'processor.ReadRequest(') p.write_text(s) PY git diff -- pipeline/workflow.go && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.LoadInput", "to": "ReadRequest" }`
4	✓ ✓ `/bin/zsh -lc "python3 - <<'PY' from pathlib import Path p = Path('pipeline/workflow.go') s = p.read_text() s = s.replace('func (Processor) CheckInput(', 'func (Processor) ValidateRequest(') s = s.replace('processor.CheckInput(', 'processor.ValidateRequest(') p.write_text(s) PY git diff -- pipeline/workflow.go && \ go test ./..."`	✓ ✓ `/bin/zsh -lc 'go test ./...'`
5	✓ ✓ `/bin/zsh -lc "python3 - <<'PY' from pathlib import Path p = Path('pipeline/workflow.go') s = p.read_text() s = s.replace('func (Processor) NormalizeValue(', 'func (Processor) CanonicalizeValue(') s = s.replace('processor.NormalizeValue(', 'processor.CanonicalizeValue(') s = s.replace('func (Processor) MapFields(', 'func (Processor) ProjectFields(') s = s.replace('processor.MapFields(', 'processor.ProjectFields(') p.write_text(s) PY git diff -- pipeline/workflow.go && \ go test ./..."`	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CheckInput" }`
6	✓ ✓ `/bin/zsh -lc "python3 - <<'PY' from pathlib import Path p = Path('pipeline/workflow.go') s = p.read_text() for old, new in [('RouteValue', 'DispatchValue'), ('StampEvent', 'AnnotateEvent'), ('CountItems', 'TallyItems'), ('MergeParts', 'CombineParts')]: s = s.replace(f'func (Processor) {old}(', f'func (Processor) {new}(') s = s.replace(f'processor.{old}(', f'processor.{new}(') p.write_text(s) PY git diff -- pipeline/workflow.go && \ go test ./..."`	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CheckInput", "to": "ValidateRequest" }`
7	✓ ✓ /bin/zsh -lc "python3 - <<'PY' from pathlib import Path p = Path('pipeline/workflow.go') s = p.read_text() for old, new in [('FilterNoise', 'PruneNoise'), ('SortRecords', 'OrderRecords'), ('EncodePayload', 'MarshalPayload'), ('DecodeReply', 'ParseReply'), ('OpenChannel', 'ConnectChannel'), ('CloseChannel', 'ReleaseChannel'), ('FlushBuffer', 'DrainBuffer'), ('SaveResult', 'PersistResult')]: s = s.replace(f'func (Processor) {old}(', f'func (Processor) {new}(') s = s.replace(f'processor.{old}(', f'processor.{new}(') p.write_text(s) PY git diff -- pipeline/workflow.go && \ go test ./..."	✓ ✓ `/bin/zsh -lc 'go test ./...'`
8	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.NormalizeValue" }`
9	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.NormalizeValue", "to": "CanonicalizeValue" }`
10	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MapFields" }`
11	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MapFields", "to": "ProjectFields" }`
12	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
13	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.RouteValue" }`
14	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.StampEvent" }`
15	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CountItems" }`
16	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.MergeParts" }`
17	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.RouteValue", "to": "DispatchValue" }`
18	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.StampEvent", "to": "AnnotateEvent" }`
19	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CountItems", "to": "TallyItems" }`
20	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.MergeParts", "to": "CombineParts" }`
21	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`
22	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FilterNoise" }`
23	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SortRecords" }`
24	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.EncodePayload" }`
25	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.DecodeReply" }`
26	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.OpenChannel" }`
27	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.CloseChannel" }`
28	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.FlushBuffer" }`
29	—	`semedit/semantic_lookup` (✓ ✓) `{ "language": "go", "symbol": "Processor.SaveResult" }`
30	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FilterNoise", "to": "PruneNoise" }`
31	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SortRecords", "to": "OrderRecords" }`
32	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.EncodePayload", "to": "MarshalPayload" }`
33	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.DecodeReply", "to": "ParseReply" }`
34	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.OpenChannel", "to": "ConnectChannel" }`
35	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.CloseChannel", "to": "ReleaseChannel" }`
36	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.FlushBuffer", "to": "DrainBuffer" }`
37	—	`semedit/semantic_rename` (✓ ✓) `{ "file": "pipeline/workflow.go", "language": "go", "symbol": "Processor.SaveResult", "to": "PersistResult" }`
38	—	✓ ✓ `/bin/zsh -lc 'go test ./...'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Benchmark run luna-medium-staged-followup-smoke-20260929 Benchmark run luna-medium-staged-prewarm-smoke-20260929

Benchmark run luna-medium-staged-prewarm-r10-20260929

Interactive browser

Baseline vs semedit

Detailed observations

Benchmark Methodology & Transparency

Test case: task-13-rename-followup-scale

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

Test case: `task-13-rename-followup-scale`

Target: `codex/gpt-6-luna/medium`