Benchmark run luna-medium-edit-scale-distinct-r10-20260928T223304Z

This page contains exactly the publishable benchmark observations recorded in run luna-medium-edit-scale-distinct-r10-20260928T223304Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

• Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
• Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
• Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
  1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
  2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
  3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
  4. Level 4 (Test Suite): Unit and integration test suite execution.
• Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-00-hi-overhead

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not modify any workspace files.

Reply hi. Do not inspect or modify the workspace. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

Test case: task-12-rename-scale-01

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
cat rename-plan.md && \
rg -n \"Processor|processor\" --glob '*.go'"

/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -80"

/bin/zsh -lc 'cat pipeline/workflow.go && \
git status --short'

/bin/zsh -lc 'cat rename-plan.md && \
cat go.mod'

/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); t=p.with_suffix(\".go.tmp\"); t.write_text(s); t.replace(p)' && \
go test ./..."

{
  "language": "go",
  "symbol": "(*Processor).LoadInput"
}

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g 'go.mod' -g 'AGENTS*'"

/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"

/bin/zsh -lc 'cat rename-plan.md && \
cat pipeline/workflow.go'

/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'AGENTS.md' -g 'AGENTS.override.md'"

/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\"); tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s); tmp.replace(p)' && \
go test ./..."

{
  "language": "go",
  "symbol": "LoadInput"
}

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'git diff --'

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
cat rename-plan.md && \
rg -n \"type Processor|Processor|func \\(.*Processor\" --glob '*.go' ."

/bin/zsh -lc "cat rename-plan.md && \
rg --files -g '*.go' -g 'go.mod' | head -80"

/bin/zsh -lc "rg --files -g '*.go' -g '"'!**/*_test.go'"' | xargs rg -n \"LoadInput|ReadRequest|Processor\""

{
  "language": "go",
  "symbol": "LoadInput"
}

/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('pipeline/workflow.go')
s = p.read_text()
s = s.replace('LoadInput', 'ReadRequest')
tmp = p.with_suffix('.go.tmp')
tmp.write_text(s)
tmp.replace(p)
PY
go test ./..."

{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"

/bin/zsh -lc 'cat rename-plan.md'

/bin/zsh -lc 'cat rename-plan.md && \
cat pipeline/workflow.go && \
cat go.mod'

/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"

/bin/zsh -lc "rg -n 'LoadInput|ReadRequest|Processor' --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"' ."

/bin/zsh -lc 'rg --files'

/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os

path = Path('pipeline/workflow.go')
text = path.read_text()
updated = text.replace('LoadInput', 'ReadRequest')
if updated == text:
    raise SystemExit('expected rename target not found')
tmp = path.with_name(path.name + '.tmp')
with tmp.open('w') as f:
    f.write(updated)
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, path)
PY
go test ./..."

/bin/zsh -lc 'cat AGENTS.override.md'

{
  "dry_run": true,
  "file": "pipeline/workflow_test.go",
  "mode": "relax"
}

{
  "symbol": "LoadInput"
}

{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g 'AGENTS*' | head -200"

/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g '*.go' | head -80 && \
git status --short"

/bin/zsh -lc 'cat rename-plan.md && \
cat pipeline/workflow.go && \
git status --short'

/bin/zsh -lc 'cat rename-plan.md && \
cat pipeline/workflow.go && \
rg --files | head -80'

{
  "language": "go",
  "symbol": "Processor.LoadInput"
}

/bin/zsh -lc 'go test ./...'

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go go.mod rename-plan.md && \
go test ./...'

/bin/zsh -lc "cat pipeline/workflow.go && \
rg 'LoadInput|ReadRequest' . --glob '"'!rename-plan.md'"' && \
git status --short -- pipeline/workflow.go pipeline/workflow_test.go go.mod rename-plan.md"

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g '"'!vendor'"'"

/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' | head -80"

/bin/zsh -lc "cat rename-plan.md && \
cat pipeline/workflow.go && \
cat go.mod && \
rg -n 'Processor|\\.\\w+\\(' --glob '*.go'"

{
  "language": "go",
  "symbol": "LoadInput"
}

/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"pipeline/workflow.go\"); data=p.read_text(); updated=data.replace(\"LoadInput\", \"ReadRequest\"); assert updated "'!= data; st=p.stat(); fd,tmp=tempfile.mkstemp(dir=p.parent, prefix=".workflow.go."); f=os.fdopen(fd,"w"); f.write(updated); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p); os.utime(p, ns=(st.st_atime_ns, max(time.time_ns(), st.st_mtime_ns+1)))'"' && \
git diff -- pipeline/workflow.go"

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'cat pipeline/workflow.go && \
go test ./...'

/bin/zsh -lc 'git diff --'

/bin/zsh -lc "git status --short && \
rg -n 'LoadInput|ReadRequest' pipeline"

/bin/zsh -lc 'git show HEAD:pipeline/workflow.go && \
git diff -- pipeline/workflow.go pipeline/workflow_test.go'

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g '*.go' | head -100"

/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -80"

/bin/zsh -lc 'cat rename-plan.md && \
cat pipeline/workflow.go && \
cat go.mod'

/bin/zsh -lc "cat rename-plan.md && \
cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md'"

/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=\"LoadInput\"; new=\"ReadRequest\"; assert s.count(old)==2; tmp=p.with_suffix(\".go.tmp\"); tmp.write_text(s.replace(old,new)); tmp.replace(p)' && \
go test ./..."

{
  "language": "go",
  "symbol": "LoadInput"
}

{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff --'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' -g '"'!rename-plan.md'"' | head -80 && \
cat rename-plan.md"

/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -80"

/bin/zsh -lc "cat pipeline/workflow.go && \
rg -n 'LoadInput|ReadRequest' . --glob '"'!rename-plan.md'"' --glob '"'!AGENTS.override.md'"'"

{
  "language": "go",
  "symbol": "LoadInput"
}

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'go test ./...'

/bin/zsh -lc 'git status --short && \
git diff -- pipeline/workflow.go pipeline/workflow_test.go go.mod rename-plan.md && \
go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "cat rename-plan.md && \
rg -n \"Processor|func \\(.*Processor|\\.\\w+\\(\" --glob '*.go' ."

/bin/zsh -lc "cat rename-plan.md && \
printf '\\n--- files ---\\n' && \
rg --files -g '*.go' -g 'go.mod' -g 'rename-plan.md'"

/bin/zsh -lc 'cat pipeline/workflow.go && \
git status --short'

{
  "symbol": "LoadInput"
}

/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); old=s; s=s.replace(\"LoadInput\", \"ReadRequest\"); assert s "'!= old; tmp=p.with_suffix(".go.tmp"); tmp.write_text(s); tmp.replace(p)'"' && \
go test ./..."

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'git diff --'

/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go && \
go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (11 lines -> 11 lines)

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'go.mod' | head -200 && \
git status --short"

/bin/zsh -lc "cat rename-plan.md && \
printf '\\n--- files ---\\n' && \
rg --files -g '"'!AGENTS.override.md'"' | head -120"

/bin/zsh -lc "cat rename-plan.md && \
cat pipeline/workflow.go && \
rg -n 'Processor|\\.Process|Process\\(' --glob '*.go' ."

{
  "language": "go",
  "symbol": "LoadInput"
}

/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('pipeline/workflow.go')
data = p.read_bytes()
old = b'LoadInput'
new = b'ReadRequest'
if data.count(old) "'!= 2:
    raise SystemExit(f'"'expected 2 occurrences, found {data.count(old)}')
updated = data.replace(old, new)
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.workflow.go.')
try:
    with os.fdopen(fd, 'wb') as f:
        f.write(updated)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
go test ./..."

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

/bin/zsh -lc 'git diff -- pipeline/workflow.go pipeline/workflow_test.go rename-plan.md go.mod && \
go test ./...'

/bin/zsh -lc 'git status --short && \
cat pipeline/workflow.go && \
cat pipeline/workflow_test.go'

Baseline vs Semedit Agentic in Large Context

Test case: task-12-rename-scale-02

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	return value
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (13 lines -> 13 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (13 lines -> 13 lines)

/bin/zsh -lc "cat rename-plan.md && \
rg -n \"type Processor|Processor|func .*Processor|\\.Process|Process\\(\" --glob '*.go'"

/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g 'go.mod' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' | head -80"

/bin/zsh -lc 'cat pipeline/workflow.go && \
rg -n "LoadInput|CheckInput|ReadRequest|ValidateRequest" .'

/bin/zsh -lc "cat rename-plan.md && \
rg -n 'Processor|func .*Processor' --glob '*.go' ."

/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"pipeline/workflow.go\"); s=p.read_text(); s=s.replace(\"LoadInput\", \"ReadRequest\").replace(\"CheckInput\", \"ValidateRequest\"); p.write_text(s)' && \
go test ./..."

{
  "language": "go",
  "symbol": "Processor.CheckInput"
}

{
  "language": "go",
  "symbol": "Processor.LoadInput"
}

{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}

{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}

/bin/zsh -lc 'git diff --'

/bin/zsh -lc 'git status --short'

/bin/zsh -lc 'go test ./...'

Baseline vs Semedit Agentic in Large Context

default prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply every Processor method rename in rename-plan.md and update all call sites. Preserve behavior and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	return value
}