Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run luna-medium-five-rename-multifile-r10-20260929T1134Z

Benchmark run luna-medium-five-rename-multifile-r10-20260929T1134Z

This page contains exactly the publishable benchmark observations recorded in run luna-medium-five-rename-multifile-r10-20260929T1134Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-12-rename-scale-05-multifile

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.26s50.94s+74.1%———
Process Start → First Event1.07s0.17s————
First Event → First Tool Call6.46s7.92s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles46+50.0%———
Initial Load / Discovery Turns10-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)46+50.0%———
Output Tokens599718+19.9%———
Reasoning / Thinking Tokens61205+236.1%———
Total Input Tokens101115166320+64.5%———
Cached Input Tokens85760143872+67.8%———
Uncached Input Tokens1535522448+46.2%———
Cost0.06810.1036+52.2%———
Cached vs Uncached Token Ratio5.59:16.41:1+14.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'pwd && \
rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'AGENTS.md' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg -n \"[A-Za-z_]\" --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"' --glob '"'!AGENTS.override.md'"' ."
✓ ✓
/bin/zsh -lc 'cat rename-plan.md && \
git status --short'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in sorted(Path('pipeline').glob('*.go')):
    if path.name.endswith('_test.go'):
        continue
    original = path.read_text()
    updated = original
    for old, new in replacements.items():
        updated = updated.replace(old, new)
    if updated == original:
        continue
    fd, temporary = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
    try:
        with os.fdopen(fd, 'w') as stream:
            stream.write(updated)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)
PY
rg -n \"LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue\" pipeline --glob '"'!**/*_test.go'"'
git diff -- pipeline"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "language": "go",
        "symbol": "Processor.LoadInput",
        "to": "ReadRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.CheckInput",
        "to": "ValidateRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.NormalizeValue",
        "to": "CanonicalizeValue"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.MapFields",
        "to": "ProjectFields"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.RouteValue",
        "to": "DispatchValue"
      },
      "tool": "semantic_rename"
    }
  ]
}
5—✓ ✓
/bin/zsh -lc 'git diff -- pipeline && \
git status --short -- pipeline rename-plan.md go.mod'
6—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency47.64s78.92s+65.6%———
Process Start → First Event0.14s0.13s————
First Event → First Tool Call12.15s5.09s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles713+85.7%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)713+85.7%———
Output Tokens973690-29.1%———
Reasoning / Thinking Tokens183102-44.3%———
Total Input Tokens170693200389+17.4%———
Cached Input Tokens152576174336+14.3%———
Uncached Input Tokens1811726053+43.8%———
Cost0.09790.1186+21.2%———
Cached vs Uncached Token Ratio8.42:16.69:1-20.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*test*'"' -g 'AGENTS.md' -g 'rename-plan.md' -g 'go.mod' | head -100"
✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
printf '\\n--- files ---\\n' && \
rg --files -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100"
2✓ ✓
/bin/zsh -lc 'cat rename-plan.md && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go'
3✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue' --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "LoadInput"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in sorted(Path('pipeline').glob('*.go')):
    if path.name.endswith('_test.go'):
        continue
    original = path.read_text()
    updated = original
    for old, new in replacements.items():
        updated = updated.replace(old, new)
    if updated "'!= original:
        temp = path.with_suffix(path.suffix + '"'.tmp')
        temp.write_text(updated)
        temp.replace(path)
PY
rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue' --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "CheckInput"
}
5✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "NormalizeValue"
}
6✓ ✓
/bin/zsh -lc 'git diff -- pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go && \
git status --short -- pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go go.mod rename-plan.md pipeline/*_test.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "MapFields"
}
7? ✗
/bin/zsh -lc 'git status --short --untracked-files=no | head -50 && \
git show HEAD:pipeline/processor.go && \
cat pipeline/processor.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "RouteValue"
}
8—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": false,
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
9—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": false,
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
10—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": false,
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
11—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": false,
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
12—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": false,
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
13—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency30.58s63.74s+108.5%———
Process Start → First Event1.07s0.17s————
First Event → First Tool Call6.01s6.05s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles59+80.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)59+80.0%———
Output Tokens963566-41.2%———
Reasoning / Thinking Tokens16189-44.7%———
Total Input Tokens122083148284+21.5%———
Cached Input Tokens109056133376+22.3%———
Uncached Input Tokens1302714908+14.4%———
Cost0.07390.0788+6.7%———
Cached vs Uncached Token Ratio8.37:18.95:1+6.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg --files -g '*.go' -g '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'AGENTS*.md' -g '*.go' -g 'go.mod' | head -100"
2✓ ✓
/bin/zsh -lc "rg -n '\\b(LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue)\\b' --glob '*.go'"
? ✗
/bin/zsh -lc "cat rename-plan.md && \
git status --short && \
rg -n 'semantic_rename|semantic_batch' . --glob '"'!go.mod'"' --glob '"'!rename-plan.md'"'"
3✓ ✓
/bin/zsh -lc 'git status --short && \
cat pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os

root = Path('.')
renames = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in sorted(root.glob('pipeline/*.go')):
    if path.name.endswith('_test.go'):
        continue
    original = path.read_text()
    updated = original
    for old, new in renames.items():
        updated = updated.replace(old, new)
    if updated == original:
        continue
    temp = path.with_name(path.name + '.tmp')
    with temp.open('w') as f:
        f.write(updated)
        f.flush()
        os.fsync(f.fileno())
    os.replace(temp, path)
PY
rg -n '\\b(LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue)\\b' pipeline --glob '*.go'"
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
5✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
6—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
7—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
8—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
9—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency27.97s69.32s+147.8%———
Process Start → First Event1.07s0.14s————
First Event → First Tool Call6.18s5.38s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles411+175.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)411+175.0%———
Output Tokens623862+38.4%———
Reasoning / Thinking Tokens32170+431.2%———
Total Input Tokens98892296784+200.1%———
Cached Input Tokens83712270592+223.2%———
Uncached Input Tokens1518026192+72.5%———
Cost0.06710.1460+117.7%———
Cached vs Uncached Token Ratio5.51:110.33:1+87.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' -g '"'!rename-plan.md'"' | head -200 && \
cat rename-plan.md"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'AGENTS*.md' -g 'go.mod' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'rg -n "LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue" pipeline'
✓ ✓
/bin/zsh -lc 'cat rename-plan.md && \
git status --short'
3file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Processor.LoadInput"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
5—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
6—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
7—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
8—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
9—✓ ✓
/bin/zsh -lc 'go test ./...'
10—✓ ✓
/bin/zsh -lc 'git diff -- pipeline && \
git status --short -- pipeline rename-plan.md go.mod'
11—✓ ✓
/bin/zsh -lc "rg -n 'ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue|LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue' pipeline"
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency28.29s54.98s+94.4%———
Process Start → First Event0.20s0.17s————
First Event → First Tool Call6.49s8.88s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles59+80.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)59+80.0%———
Output Tokens623556-10.8%———
Reasoning / Thinking Tokens090————
Total Input Tokens117229167840+43.2%———
Cached Input Tokens90624126464+39.5%———
Uncached Input Tokens2660541376+55.5%———
Cost0.09700.1431+47.6%———
Cached vs Uncached Token Ratio3.41:13.06:1-10.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg --files -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc 'cat rename-plan.md'
2✓ ✓
/bin/zsh -lc "rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue' --glob '*.go'"
semedit/semantic_assertion_mode (✓ ✗): param "file" is required: invalid operation parameters
{
  "dry_run": true,
  "file": "",
  "mode": "relax"
}
3✓ ✓
/bin/zsh -lc 'cat pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "LoadInput"
}
4file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "CheckInput"
}
5✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "NormalizeValue"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "MapFields"
}
7—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "RouteValue"
}
8—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "language": "go",
        "symbol": "Processor.LoadInput",
        "to": "ReadRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.CheckInput",
        "to": "ValidateRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.NormalizeValue",
        "to": "CanonicalizeValue"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.MapFields",
        "to": "ProjectFields"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.RouteValue",
        "to": "DispatchValue"
      },
      "tool": "semantic_rename"
    }
  ]
}
9—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency34.28s92.24s+169.1%———
Process Start → First Event0.30s0.17s————
First Event → First Tool Call6.86s5.65s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles419+375.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)419+375.0%———
Output Tokens9961296+30.1%———
Reasoning / Thinking Tokens98335+241.8%———
Total Input Tokens99492266973+168.3%———
Cached Input Tokens87808227072+158.6%———
Uncached Input Tokens1168439901+241.5%———
Cost0.06480.1769+172.9%———
Cached vs Uncached Token Ratio7.52:15.69:1-24.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' | head -80 && \
cat rename-plan.md"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc "rg -n \"LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue\" . -g '"'!rename-plan.md'"' -g '"'!AGENTS.override.md'"' && \
cat pipeline/processor.go && \
cat pipeline/workflow.go && \
cat pipeline/stage_input.go && \
cat pipeline/stage_validation.go && \
cat pipeline/stage_normalization.go && \
cat pipeline/stage_mapping.go && \
cat pipeline/stage_routing.go"
✓ ✓
/bin/zsh -lc 'cat rename-plan.md'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile

root = Path('.')
renames = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
files = [
    root / 'pipeline/processor.go',
    root / 'pipeline/stage_input.go',
    root / 'pipeline/stage_validation.go',
    root / 'pipeline/stage_normalization.go',
    root / 'pipeline/stage_mapping.go',
    root / 'pipeline/stage_routing.go',
]
for path in files:
    original = path.read_text()
    updated = original
    for old, new in renames.items():
        updated = updated.replace(old, new)
    if updated "'!= original:
        fd, name = tempfile.mkstemp(dir=path.parent, prefix=f'"'.{path.name}.')
        try:
            with os.fdopen(fd, 'w') as stream:
                stream.write(updated)
                stream.flush()
                os.fsync(stream.fileno())
            os.replace(name, path)
        except Exception:
            os.unlink(name)
            raise
PY
rg -n \"LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue\" pipeline -g '"'!*.test.go'"'"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md'
4✓ ✓
/bin/zsh -lc 'go test ./...'
? ✗
/bin/zsh -lc 'rg -n "semantic_(lookup|rename|batch|verify|inspect_symbol)|mcp.*semantic" .'
5—semedit/semantic_assertion_mode (✓ ✓)
{
  "dry_run": true,
  "file": "pipeline/workflow.go",
  "mode": "relax"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
7—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
8—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
9—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
10—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
11—semedit/semantic_find_references (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
12—semedit/semantic_find_references (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
13—semedit/semantic_find_references (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
14—semedit/semantic_find_references (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
15—semedit/semantic_find_references (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
16—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "language": "go",
        "symbol": "Processor.LoadInput",
        "to": "ReadRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.CheckInput",
        "to": "ValidateRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.NormalizeValue",
        "to": "CanonicalizeValue"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.MapFields",
        "to": "ProjectFields"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.RouteValue",
        "to": "DispatchValue"
      },
      "tool": "semantic_rename"
    }
  ]
}
17—? ✗
/bin/zsh -lc "rg -n \"LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue\" --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"' --glob '"'!AGENTS.override.md'"' ."
18—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --name-only'
19—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency32.76s37.27s+13.8%———
Process Start → First Event0.49s0.18s————
First Event → First Tool Call6.79s6.36s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles54-20.0%———
Initial Load / Discovery Turns10-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)54-20.0%———
Output Tokens729581-20.3%———
Reasoning / Thinking Tokens89152+70.8%———
Total Input Tokens122992118371-3.8%———
Cached Input Tokens107008101120-5.5%———
Uncached Input Tokens1598417251+7.9%———
Cost0.07690.0776+0.8%———
Cached vs Uncached Token Ratio6.69:15.86:1-12.4%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'pwd && \
rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg -n \"[A-Za-z_]\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat rename-plan.md && \
git status --short'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in [Path('pipeline/processor.go'), *Path('pipeline').glob('stage_*.go')]:
    original = path.read_text()
    updated = original
    for old, new in replacements.items():
        updated = updated.replace(old, new)
    if updated == original:
        continue
    fd, temporary = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as output:
            output.write(updated)
            output.flush()
            os.fsync(output.fileno())
        os.replace(temporary, path)
    except BaseException:
        try:
            os.unlink(temporary)
        except FileNotFoundError:
            pass
        raise
PY
git diff -- pipeline/processor.go pipeline/stage_input.go pipeline/stage_validation.go pipeline/stage_normalization.go pipeline/stage_mapping.go pipeline/stage_routing.go"
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "language": "go",
        "symbol": "Processor.LoadInput",
        "to": "ReadRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.CheckInput",
        "to": "ValidateRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.NormalizeValue",
        "to": "CanonicalizeValue"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.MapFields",
        "to": "ProjectFields"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "language": "go",
        "symbol": "Processor.RouteValue",
        "to": "DispatchValue"
      },
      "tool": "semantic_rename"
    }
  ]
}
4✓ ✓
/bin/zsh -lc "git status --short && \
cat pipeline/processor.go && \
cat pipeline/stage_input.go && \
rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue' pipeline"
✓ ✓
/bin/zsh -lc 'go test ./...'
5✓ ✓
/bin/zsh -lc 'go test ./...'
—
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency23.75s66.91s+181.8%———
Process Start → First Event0.17s0.14s————
First Event → First Tool Call6.25s10.27s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles49+125.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)49+125.0%———
Output Tokens619454-26.7%———
Reasoning / Thinking Tokens590-100.0%———
Total Input Tokens99063212113+114.1%———
Cached Input Tokens77568185600+139.3%———
Uncached Input Tokens2149526513+23.3%———
Cost0.08160.1184+45.0%———
Cached vs Uncached Token Ratio3.61:17.00:1+94.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'rename-plan.md' -g '*.go' -g 'go.mod'"
✓ ✓
/bin/zsh -lc 'cat rename-plan.md'
2✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg -n \"\\b[A-Za-z_][A-Za-z0-9_]*\\b\" pipeline --glob '*.go'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g '*.go' | head -80"
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for name in ['pipeline/processor.go', 'pipeline/stage_input.go', 'pipeline/stage_validation.go', 'pipeline/stage_normalization.go', 'pipeline/stage_mapping.go', 'pipeline/stage_routing.go']:
    path = Path(name)
    content = path.read_text()
    for old, new in replacements.items():
        content = content.replace(old, new)
    fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(temp_name, path)
    except BaseException:
        os.unlink(temp_name)
        raise
PY
rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue' --glob '*.go'"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "pipeline"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
5—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
6—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
7—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
8—semedit/semantic_rename (✓ ✓)
{
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
9—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.10s79.04s+171.6%———
Process Start → First Event0.14s0.14s————
First Event → First Tool Call6.94s7.00s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles413+225.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)413+225.0%———
Output Tokens726536-26.2%———
Reasoning / Thinking Tokens1140-100.0%———
Total Input Tokens97671192950+97.6%———
Cached Input Tokens64512168192+160.7%———
Uncached Input Tokens3315924758-25.3%———
Cost0.10950.1106+1.0%———
Cached vs Uncached Token Ratio1.95:16.79:1+249.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!rename-plan.md'"' | head -200 && \
cat rename-plan.md"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue' --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"
✓ ✓
/bin/zsh -lc 'cat rename-plan.md'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in sorted(Path('pipeline').glob('*.go')):
    if path.name.endswith('_test.go'):
        continue
    original = path.read_text()
    updated = original
    for old, new in replacements.items():
        updated = updated.replace(old, new)
    if updated == original:
        continue
    fd, temp_name = tempfile.mkstemp(prefix=path.name + '.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(updated)
            f.flush()
            os.fsync(f.fileno())
        old_mtime = path.stat().st_mtime_ns
        os.replace(temp_name, path)
        new_ns = max(time.time_ns(), old_mtime + 1_000_000)
        os.utime(path, ns=(new_ns, new_ns))
    finally:
        if os.path.exists(temp_name):
            os.unlink(temp_name)
PY
rg -n 'LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue' --glob '"'!rename-plan.md'"' --glob '"'!go.mod'"'"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
5—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
7—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
8—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
9—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
10—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
11—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
12—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
13—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-rename-scale-05-multifile
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_12_rename_scale_05_multifile.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Perform these 5 renames consecutively, possibly batched if you know how. Use rename-plan.md, update every call site, preserve behavior, and run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline keeps the rename targets together so the oracle can check all five declarations.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency20.40s49.53s+142.8%———
Process Start → First Event0.54s0.16s————
First Event → First Tool Call5.22s6.13s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles48+100.0%———
Initial Load / Discovery Turns00————
MCP Discovery / Schema Turns00————
Tool Invocations (per run)48+100.0%———
Output Tokens498566+13.7%———
Reasoning / Thinking Tokens36108+200.0%———
Total Input Tokens101342115811+14.3%———
Cached Input Tokens8985694976+5.7%———
Uncached Input Tokens1148620835+81.4%———
Cost0.05790.0843+45.6%———
Cached vs Uncached Token Ratio7.82:14.56:1-41.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/processor.go modified (10 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g '*.go' -g 'go.mod' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg --files -g 'AGENTS.md' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc 'cat rename-plan.md && \
rg -n "" pipeline/*.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "LoadInput"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
replacements = {
    'LoadInput': 'ReadRequest',
    'CheckInput': 'ValidateRequest',
    'NormalizeValue': 'CanonicalizeValue',
    'MapFields': 'ProjectFields',
    'RouteValue': 'DispatchValue',
}
for path in sorted(Path('pipeline').glob('*.go')):
    if path.name.endswith('_test.go'):
        continue
    text = path.read_text()
    for old, new in replacements.items():
        text = text.replace(old, new)
    path.write_text(text)
PY
rg -n \"LoadInput|CheckInput|NormalizeValue|MapFields|RouteValue|ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|DispatchValue\" pipeline"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "CheckInput"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "NormalizeValue"
}
5—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "RouteValue"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "MapFields"
}
7—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "symbol": "Processor.LoadInput",
        "to": "ReadRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "symbol": "Processor.CheckInput",
        "to": "ValidateRequest"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "symbol": "Processor.NormalizeValue",
        "to": "CanonicalizeValue"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "symbol": "Processor.MapFields",
        "to": "ProjectFields"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "symbol": "Processor.RouteValue",
        "to": "DispatchValue"
      },
      "tool": "semantic_rename"
    }
  ]
}
8—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published
Benchmark run luna-medium-edit-scale-distinct-r10-20260928T223304ZBenchmark run luna-medium-staged-followup-r10-20260929T054733Z