Skip to content
Benchmark run gpt-6-luna-all-r5-c10-none-20260926-retry

Benchmark run gpt-6-luna-all-r5-c10-none-20260926-retry

This page contains exactly the publishable benchmark observations recorded in run gpt-6-luna-all-r5-c10-none-20260926-retry; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-04-insert-public

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency17.16s25.36s+47.8%13.88s39.63s+185.5%
Process Start → First Event1.14s1.14s—0.55s0.18s—
First Event → First Tool Call6.12s6.41s—5.05s6.15s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles440%27+250.0%
Initial Load / Discovery Turns21-50.0%220%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)440%27+250.0%
Output Tokens361419+16.1%281776+176.2%
Reasoning / Thinking Tokens15115+666.7%0318—
Total Input Tokens95647100684+5.3%75847160498+111.6%
Cached Input Tokens8064078592-2.5%66560137472+106.5%
Uncached Input Tokens1500722092+47.2%928723026+147.9%
Cost0.06240.0816+30.7%0.04340.1056+143.5%
Cached vs Uncached Token Ratio5.37:13.56:1-33.8%7.17:15.97:1-16.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS*' -g '*.go' -g 'PROJECT_BRIEF.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'go.mod' | head -100"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat AGENTS.override.md'
? ✗
/bin/zsh -lc 'cat api/server.go && \
rg -n "semantic_insert_function|semantic_lookup" .'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
s = s.replace('func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}', 'func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}')
t = p.with_suffix('.go.tmp')
t.write_text(s)
t.replace(p)
PY
git diff -- api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4✓ ✓
/bin/zsh -lc 'cat api/server.go; git status --short'
✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g '*.go' -g 'AGENTS.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
sed -n '1,180p' api/router.go"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g 'go.mod' -g '"'!**/*_test.go'"'"
3—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "api/server.go"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "language": "go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
6—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- api/server.go'
7—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "api/server.go"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency18.73s34.92s+86.4%16.85s30.67s+82.0%
Process Start → First Event1.61s1.37s—0.37s0.14s—
First Event → First Tool Call7.35s8.59s—5.42s5.16s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles35+66.7%440%
Initial Load / Discovery Turns21-50.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)35+66.7%440%
Output Tokens362463+27.9%504471-6.5%
Reasoning / Thinking Tokens0114—0139—
Total Input Tokens76375167241+119.0%76697112768+47.0%
Cached Input Tokens54528138752+154.5%6758491904+36.0%
Uncached Input Tokens2184728489+30.4%911320864+128.9%
Cost0.07280.1131+55.4%0.04600.0828+80.0%
Cached vs Uncached Token Ratio2.50:14.87:1+95.1%7.42:14.40:1-40.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g '"'!vendor'"' | head -80 && \
git status --short"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g '*.go' -g '"'!**/*_test.go'"'"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat api/server_test.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Server"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
old = 'func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
new = 'func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n'
if old not in s:
    raise SystemExit('expected insertion point not found')
p.write_text(s.replace(old, new, 1))
PY
cat api/server.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
4—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg -n "type Server|func .*Server|''^func " --glob '"'*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100"
2✓ ✓
/bin/zsh -lc 'cat api/server_test.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
3✓ ✓
/bin/zsh -lc 'cat api/server.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"api/server.go\"); s=p.read_text(); old=\"type Server struct {\\n\\tPort int\\n}\\n\\n\"; new=old+\"func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n\"; assert old in s and \"func InitServer()\" not in s; p.write_text(s.replace(old,new,1))'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "InitServer"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency20.13s58.34s+189.9%16.04s38.54s+140.3%
Process Start → First Event1.39s1.41s—0.15s0.15s—
First Event → First Tool Call6.90s15.24s—4.11s5.18s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles34+33.3%440%
Initial Load / Discovery Turns220%31-66.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)34+33.3%440%
Output Tokens486426-12.3%424575+35.6%
Reasoning / Thinking Tokens65103+58.5%72244+238.9%
Total Input Tokens76302131457+72.3%95237148453+55.9%
Cached Input Tokens68608111104+61.9%72704123392+69.7%
Uncached Input Tokens769420353+164.5%2253325061+11.2%
Cost0.04330.0853+97.0%0.08070.1037+28.5%
Cached vs Uncached Token Ratio8.92:15.46:1-38.8%3.23:14.92:1+52.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat go.mod && \
git status --short'
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"api/server.go\"); old=p.read_text(); needle=\"func (s *Server) internalRun() {}\\n\"; assert old.count(needle)==1; new=old.replace(needle, \"func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n\"+needle); fd, tmp=tempfile.mkstemp(dir=str(p.parent)); f=os.fdopen(fd,\"w\"); f.write(new); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p); os.utime(p, ns=(time.time_ns(), time.time_ns()))'"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
4—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*server*.go' -g 'README*' | head -80"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go"
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
3✓ ✓
/bin/zsh -lc "sed -n '1,200p' api/router.go && \
sed -n '1,160p' config/config.go && \
sed -n '1,180p' api/server_test.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); old=p.read_text(); needle=\"func (s *Server) Start() {}\\n\\n\"; assert needle in old; new=old.replace(needle, needle+\"func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n\", 1); fd, tmp=tempfile.mkstemp(dir=p.parent); f=os.fdopen(fd,\"w\"); f.write(new); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)'"
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency17.86s27.31s+52.9%19.40s28.93s+49.1%
Process Start → First Event1.62s1.60s—0.14s0.15s—
First Event → First Tool Call6.55s7.20s—6.17s6.35s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles24+100.0%25+150.0%
Initial Load / Discovery Turns220%23+50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)24+100.0%25+150.0%
Output Tokens306291-4.9%269544+102.2%
Reasoning / Thinking Tokens720-100.0%63169+168.3%
Total Input Tokens7619788614+16.3%75698118273+56.2%
Cached Input Tokens6246462720+0.4%6246494976+52.0%
Uncached Input Tokens1373325894+88.6%1323423297+76.0%
Cost0.05470.0841+53.7%0.05290.0909+72.0%
Cached vs Uncached Token Ratio4.55:12.42:1-46.7%4.72:14.08:1-13.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g 'README*' | head -80 && \
git status --short"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'README.md'"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat api/server_test.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md; if [ -f PROJECT_BRIEF.md ]; then cat PROJECT_BRIEF.md; fi'
3—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "api/server.go"
}
4—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g '"'!vendor'"' | head -120"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go"
✓ ✓
/bin/zsh -lc "rg -n \"type Server struct|func [A-Z]|func [a-z]\" --glob '*.go' | head -180"
3—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat api/server_test.go && \
rg --files | head -100'
4—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
5—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "api/server.go"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency20.41s25.11s+23.0%23.58s47.45s+101.2%
Process Start → First Event1.48s1.45s—0.17s0.30s—
First Event → First Tool Call5.98s6.54s—4.92s4.92s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%21-50.0%
Internal Tool Cycles43-25.0%38+166.7%
Initial Load / Discovery Turns32-33.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)43-25.0%38+166.7%
Output Tokens438275-37.2%8041156+43.8%
Reasoning / Thinking Tokens8614-83.7%94389+313.8%
Total Input Tokens95359100040+4.9%171238230060+34.4%
Cached Input Tokens7961689856+12.9%152320202240+32.8%
Uncached Input Tokens1574310184-35.3%1891827820+47.1%
Cost0.06580.0515-21.7%0.09660.1394+44.3%
Cached vs Uncached Token Ratio5.06:18.82:1+74.5%8.05:17.27:1-9.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "rg --files -g '"'!*test*'"' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'README*' | head -120"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
3✓ ✓
/bin/zsh -lc 'cat api/server_test.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); data=p.read_text(); old=\"func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n\"; new=\"func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n\"; assert old in data; fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".server.go.\"); f=os.fdopen(fd,\"w\"); f.write(data.replace(old,new)); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)'"
—
Baseline vs Semedit Agentic in Large Context
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go && \
cat api/router.go'
3✓ ✓
/bin/zsh -lc "git status --short && \
sed -n '1,160p' api/server.go && \
git diff -- go.mod go.sum"
semedit/semantic_insert_function (✓ ✗): calculate function offset: section placement violation: strategy "private_start", target symbol "InitServer", in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-04-insert-public_1790455790112087000/api/server.go
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "private_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4—? ✗
/bin/zsh -lc 'cat docs/SUBOPTIMAL_TOOLS.md'
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
6—✓ ✓
/bin/zsh -lc 'rg --files | sort | head -60 && \
git diff -- api/server.go'
7—✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); p.parent.mkdir(parents=True, exist_ok=True); data=\"| Tool | Target | Failure | Workaround | Root cause |\\n|---|---|---|---|---|\\n| "'`semantic_insert_function` | `api/server.go` | `private_start` rejected with a section placement violation because the requested function is exported. | Retried with `before_symbol` anchored at `internalRun`; insertion succeeded. | Public functions must be placed in the public section before private helpers. |'"\\n\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)' && \
git diff -- api/server.go docs/SUBOPTIMAL_TOOLS.md"
8—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat docs/SUBOPTIMAL_TOOLS.md && \
git status --short'

Test case: task-07-generate-template-main

Target: codex/gpt-6-luna/medium

crypto_rand prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency19.78s43.06s+117.7%———
Process Start → First Event0.34s0.29s————
First Event → First Tool Call5.50s5.93s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens708614-13.3%———
Reasoning / Thinking Tokens136215+58.1%———
Total Input Tokens76548131059+71.2%———
Cached Input Tokens66560114176+71.5%———
Uncached Input Tokens998816883+69.0%———
Cost0.05220.0811+55.5%———
Cached vs Uncached Token Ratio6.66:16.76:1+1.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'SUBOPTIMAL_TOOLS.md' | head -50 && \
sed -n '1,160p' main.go"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\ntmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tpanic(err)\n}\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"main.go\"); data=\"\"\"package main

import (
\\t\\\"crypto/rand\\\"
\\t\\\"math/big\\\"
\\t\\\"os\\\"
\\t\\\"text/template\\\"
)

func main() {
\\tn, err := rand.Int(rand.Reader, big.NewInt(100))
\\tif err "'!= nil {
'"\\t\\tpanic(err)
\\t}

\\ttmpl := template.Must(template.New(\\\"greeting\\\").Parse(\"Hello World {{.}}\\\\n\"))
\\tif err := tmpl.Execute(os.Stdout, n); err "'!= nil {
'"\\t\\tpanic(err)
\\t}
}
\"\"\"; fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \
gofmt -w main.go && \
cat main.go && \
git status --short"
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "crypto/rand",
    "math/big",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency22.29s30.33s+36.0%———
Process Start → First Event0.14s0.14s————
First Event → First Tool Call5.17s6.71s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles35+66.7%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)35+66.7%———
Output Tokens656453-30.9%———
Reasoning / Thinking Tokens1800-100.0%———
Total Input Tokens76389151859+98.8%———
Cached Input Tokens66560132608+99.2%———
Uncached Input Tokens982919251+95.9%———
Cost0.05170.0869+68.3%———
Cached vs Uncached Token Ratio6.77:16.89:1+1.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "main.go"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time
p = Path('main.go')
data = '''package main

import (
\\t\"crypto/rand\"
\\t\"math/big\"
\\t\"os\"
\\t\"text/template\"
)

func main() {
\\tn, err := rand.Int(rand.Reader, big.NewInt(100))
\\tif err "'!= nil {
'"\\t\\tpanic(err)
\\t}

\\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
\\tif err := tmpl.Execute(os.Stdout, n.Int64()); err "'!= nil {
'"\\t\\tpanic(err)
\\t}
}
'''
fd, name = tempfile.mkstemp(prefix='.main.go.', dir=p.parent)
try:
    with os.fdopen(fd, 'w') as f:
        f.write(data)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
    now = time.time_ns()
    os.utime(p, ns=(now, now))
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
cat main.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nvar value [8]byte\nif _, err := rand.Read(value[:]); err != nil {\n\tpanic(err)\n}\nnumber := binary.LittleEndian.Uint64(value[:])\nif err := tmpl.Execute(os.Stdout, number); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency12.44s37.20s+199.1%———
Process Start → First Event0.15s0.15s————
First Event → First Tool Call4.01s6.45s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles16+500.0%———
Initial Load / Discovery Turns12+100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)16+500.0%———
Output Tokens327723+121.1%———
Reasoning / Thinking Tokens0244————
Total Input Tokens56752193568+241.1%———
Cached Input Tokens36352175104+381.7%———
Uncached Input Tokens2040018464-9.5%———
Cost0.06420.1020+59.0%———
Cached vs Uncached Token Ratio1.78:19.48:1+432.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'README*' -g 'PROJECT_BRIEF.md'"
2—✓ ✓
/bin/zsh -lc 'cat main.go'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\nt := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := t.Execute(os.Stdout, n); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
6—✓ ✓
/bin/zsh -lc 'go run . && \
git status --short'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency19.73s24.75s+25.5%———
Process Start → First Event0.16s0.14s————
First Event → First Tool Call4.96s4.97s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles440%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)440%———
Output Tokens614431-29.8%———
Reasoning / Thinking Tokens1610-100.0%———
Total Input Tokens95811136856+42.8%———
Cached Input Tokens85760120320+40.3%———
Uncached Input Tokens1005116536+64.5%———
Cost0.05630.0768+36.5%———
Cached vs Uncached Token Ratio8.53:17.28:1-14.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'README*' -g 'AGENTS*' -g 'go.mod' -g 'go.sum' -g 'SUBOPTIMAL_TOOLS.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat main.go'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
p = Path('main.go')
tmp = p.with_name('main.go.tmp')
content = '''package main

import (
	\"crypto/rand\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(100))
	if err "'!= nil {
		panic(err)
	}

	tmpl := template.Must(template.New("greeting").Parse("Hello World {{.}}'"\\\\n\"))
	if err := tmpl.Execute(os.Stdout, n); err "'!= nil {
		panic(err)
	}
}
'"'''
with tmp.open('w') as f:
    f.write(content)
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, p)
PY
gofmt -w main.go
git diff -- main.go go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(1\u003c\u003c31))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\"))\n\tif err := tmpl.Execute(os.Stdout, n.Int64()); err != nil {\n\t\tpanic(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat main.go && \
go build ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency16.82s46.34s+175.5%———
Process Start → First Event0.14s0.14s————
First Event → First Tool Call4.74s12.09s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles27+250.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)27+250.0%———
Output Tokens530499-5.8%———
Reasoning / Thinking Tokens1750-100.0%———
Total Input Tokens76308134650+76.5%———
Cached Input Tokens66560118272+77.7%———
Uncached Input Tokens974816378+68.0%———
Cost0.04980.0768+54.0%———
Cached vs Uncached Token Ratio6.83:17.22:1+5.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'AGENTS*'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat AGENTS.override.md && \
cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\ntmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
6—? ✗
/bin/zsh -lc 'make check'
7—✓ ✓
/bin/zsh -lc 'git status --short -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency17.09s39.36s+130.4%———
Process Start → First Event0.21s0.21s————
First Event → First Tool Call4.70s4.60s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles27+250.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)27+250.0%———
Output Tokens405830+104.9%———
Reasoning / Thinking Tokens113277+145.1%———
Total Input Tokens75765158770+109.6%———
Cached Input Tokens66560129280+94.2%———
Uncached Input Tokens920529490+220.4%———
Cost0.04610.1199+159.9%———
Cached vs Uncached Token Ratio7.23:14.38:1-39.4%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'README*' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat main.go'
3—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "os",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand"
  ]
}
6—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
7—✓ ✓
/bin/zsh -lc 'cat main.go && \
git diff -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.87s129.02s+398.7%———
Process Start → First Event0.14s0.15s————
First Event → First Tool Call4.93s15.29s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles48+100.0%———
Initial Load / Discovery Turns42-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)48+100.0%———
Output Tokens5871600+172.6%———
Reasoning / Thinking Tokens207676+226.6%———
Total Input Tokens115048226666+97.0%———
Cached Input Tokens99840203520+103.8%———
Uncached Input Tokens1520823146+52.2%———
Cost0.07290.1372+88.2%———
Cached vs Uncached Token Ratio6.56:18.79:1+33.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'README*' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff -- main.go && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go'
? ✗
/bin/zsh -lc 'cat docs/SUBOPTIMAL_TOOLS.md'
5—✓ ✓
/bin/zsh -lc 'rg --files'
6—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\trandomInt, err := rand.Int(rand.Reader, big.NewInt(1\u003c\u003c31))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, randomInt); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
7—✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); p.parent.mkdir(parents=True, exist_ok=True); s=\"# Suboptimal semantic tool behavior\\n\\n- Tool: "'`semantic_replace_body`'"\\n  Target: "'`main.go`'"\\n  Observed failure: automatic import resolution selected "'`crypto/rand`, while the inserted `rand.Int()` call used the `math/rand` signature, producing a Go diagnostic.'"\\n  Workaround: replaced the body with a "'`crypto/rand.Int(rand.Reader, big.NewInt(...))` call through the semantic editor.'"\\n  Root cause: ambiguous standard-library package name "'`rand` was resolved differently than the call assumed.'"\\n\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".suboptimal.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p); os.utime(p,None)' && \
go run . && \
git status --short"
8—✓ ✓
/bin/zsh -lc 'git diff -- main.go docs/SUBOPTIMAL_TOOLS.md && \
git status --short --untracked-files=all | head -30'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency21.48s27.95s+30.1%———
Process Start → First Event0.15s0.15s————
First Event → First Tool Call4.98s5.24s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles45+25.0%———
Initial Load / Discovery Turns42-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)45+25.0%———
Output Tokens576389-32.5%———
Reasoning / Thinking Tokens1830-100.0%———
Total Input Tokens117046122269+4.5%———
Cached Input Tokens98816105216+6.5%———
Uncached Input Tokens1823017053-6.5%———
Cost0.07980.0738-7.5%———
Cached vs Uncached Token Ratio5.42:16.17:1+13.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat main.go'
3✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
gofmt -d main.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short --untracked-files=all'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run main.go'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency11.99s60.73s+406.4%———
Process Start → First Event0.36s0.38s————
First Event → First Tool Call4.53s6.70s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles39+200.0%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)39+200.0%———
Output Tokens2381442+505.9%———
Reasoning / Thinking Tokens0597————
Total Input Tokens76197189458+148.6%———
Cached Input Tokens66560171008+156.9%———
Uncached Input Tokens963718450+91.4%———
Cost0.04370.1144+161.7%———
Cached vs Uncached Token Ratio6.91:19.27:1+34.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'ls -la && \
cat main.go'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g 'README*'"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "cat > main.go <<'EOF'
package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))
	_ = tmpl.Execute(os.Stdout, rand.Int())
}
EOF
gofmt -w main.go
cat main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc "rg --files -g 'SUBOPTIMAL_TOOLS.md' -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'Makefile'"
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand/v2",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
7—✓ ✓
/bin/zsh -lc 'rg --files'
8—✓ ✓
/bin/zsh -lc "mkdir -p docs && \
python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); data=\"# Suboptimal Tool Behavior\\n\\n- Tool: "'`semantic_replace_body`'"\\n  Target: "'`main.go`'"\\n  Observed failure: automatic import organization selected "'`crypto/rand` and `html/template`; diagnostics showed `rand.Int()` required arguments.'"\\n  Workaround: used "'`semantic_organize_imports` to replace these with `math/rand/v2` and `text/template`.'"\\n  Root cause: import inference selected packages by matching identifiers without respecting the intended random integer API and template package.\\n\"; fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)'"
9—✓ ✓
/bin/zsh -lc 'go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency27.59s28.34s+2.7%———
Process Start → First Event0.14s0.14s————
First Event → First Tool Call5.00s5.16s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns110%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens913538-41.1%———
Reasoning / Thinking Tokens446132-70.4%———
Total Input Tokens77119130745+69.5%———
Cached Input Tokens67584114176+68.9%———
Uncached Input Tokens953516569+73.8%———
Cost0.05770.0783+35.7%———
Cached vs Uncached Token Ratio7.09:16.89:1-2.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' | sort && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g 'README*'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('main.go')
data = '''// This file provides the executable entry point for rendering a greeting template.
package main

import (
	\"log\"
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World"'! Random integer: {{.}}'"\\\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
		log.Fatal(err)
	}
}
'"'''
fd, name = tempfile.mkstemp(dir=p.parent, prefix='.main.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(data)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
sed -n '1,120p' main.go
git status --short -- main.go go.mod go.sum"
? ✗
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
rg -n "semantic_(replace_body|inspect_symbol|lookup|organize_imports|verify)|tool" .'
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git status --short -- main.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency28.31s47.97s+69.4%———
Process Start → First Event0.16s0.15s————
First Event → First Tool Call8.28s12.74s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles47+75.0%———
Initial Load / Discovery Turns41-75.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)47+75.0%———
Output Tokens8231079+31.1%———
Reasoning / Thinking Tokens409466+13.9%———
Total Input Tokens139878236841+69.3%———
Cached Input Tokens127232210944+65.8%———
Uncached Input Tokens1264625897+104.8%———
Cost0.07880.1368+73.5%———
Cached vs Uncached Token Ratio10.06:18.15:1-19.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -80"
✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'README*' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat AGENTS.override.md'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "main.go"
}
3✓ ✓
/bin/zsh -lc 'git diff -- main.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go; git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- main.go go.mod go.sum'
6—✓ ✓
/bin/zsh -lc 'go run .'
7—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency32.00s66.09s+106.5%———
Process Start → First Event0.14s0.14s————
First Event → First Tool Call13.76s4.99s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles49+125.0%———
Initial Load / Discovery Turns42-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)49+125.0%———
Output Tokens8591504+75.1%———
Reasoning / Thinking Tokens320673+110.3%———
Total Input Tokens226941262544+15.7%———
Cached Input Tokens204800234240+14.4%———
Uncached Input Tokens2214128304+27.8%———
Cost0.12130.1565+29.1%———
Cached vs Uncached Token Ratio9.25:18.28:1-10.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' | sort && \
git status --short -- main.go go.mod go.sum && \
cat main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' | head -80"
2? ✗
/bin/zsh -lc "cat go.mod && \
rg -n 'text/template|rand.Int|Hello World' . --glob '"'!go.sum'"' --glob '"'!go.mod'"'"
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
3✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short -- main.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
5—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := texttmpl.Must(texttmpl.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, mathrand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "mathrand math/rand",
    "texttmpl text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
7—semedit/report_feedback (✓ ✓)
{
  "command": "semantic_replace_body",
  "intent": "Implement a Go main function that renders a greeting with a random integer using text/template.",
  "interface": "mcp",
  "manual_touchups": "Follow-up semantic import organization was needed to select explicit package aliases.",
  "observed_result": "Automatic import resolution selected crypto/rand and html/template for ambiguous identifiers, introducing a call signature error and leaving the later explicit aliases unresolved.",
  "parameters": {
    "auto_organize_imports": true,
    "file": "main.go",
    "symbol": "main"
  },
  "target_context": "main.go",
  "unexpected_reason": "Import organization did not select the intended standard library packages for text/template and math/rand."
}
8—✓ ✓
/bin/zsh -lc 'go run main.go && \
git diff -- main.go go.mod go.sum && \
git status --short'
9—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.20s53.67s+121.7%———
Process Start → First Event0.15s0.57s————
First Event → First Tool Call9.94s6.58s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles18+700.0%———
Initial Load / Discovery Turns12+100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)18+700.0%———
Output Tokens6561234+88.1%———
Reasoning / Thinking Tokens265551+107.9%———
Total Input Tokens162711298955+83.7%———
Cached Input Tokens142848273920+91.8%———
Uncached Input Tokens1986325035+26.0%———
Cost0.09690.1534+58.3%———
Cached vs Uncached Token Ratio7.19:110.94:1+52.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'README*'"
2—? ✗
/bin/zsh -lc "cat main.go; rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile'"
3—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "\ttmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\n\t_ = tmpl.Execute(os.Stdout, rand.Intn(100))",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'cat main.go'
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
7—semedit/semantic_replace_body (✓ ✓)
{
  "body": "\ttmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\t\tpanic(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
8—✓ ✓
/bin/zsh -lc "rg --files -g 'SUBOPTIMAL_TOOLS.md' -g 'Makefile' -g 'PROJECT_BRIEF.md'; cat main.go; git status --short"
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency34.59s63.03s+82.2%———
Process Start → First Event0.36s0.15s————
First Event → First Tool Call11.04s14.38s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles35+66.7%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)35+66.7%———
Output Tokens926752-18.8%———
Reasoning / Thinking Tokens344239-30.5%———
Total Input Tokens120903202016+67.1%———
Cached Input Tokens109056178432+63.6%———
Uncached Input Tokens1184723584+99.1%———
Cost0.07280.1160+59.4%———
Cached vs Uncached Token Ratio9.21:17.57:1-17.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'main.go' -g 'AGENTS*' | head -60"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc "cat main.go && \
rg --files -g 'go.mod' -g 'go.sum' -g '*.md'"
3✓ ✓
/bin/zsh -lc "python3 -c 'import os,tempfile,time; p=\"main.go\"; s=\"package main\\n\\nimport (\\n\\t\\\"math/rand\\\"\\n\\t\\\"os\\\"\\n\\t\\\"text/template\\\"\\n)\\n\\nfunc main() {\\n\\ttmpl := template.Must(template.New(\\\"greeting\\\").Parse(\\\"Hello World {{.}}\\\\n\\\"))\\n\\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {'"\\n\\t\\tpanic(err)\\n\\t}\\n}\\n\"; st=os.stat(p); fd,tmp=tempfile.mkstemp(dir=\".\",prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.utime(tmp,ns=(st.st_atime_ns,max(time.time_ns(),st.st_mtime_ns+1))); os.replace(tmp,p)' && \
gofmt -w main.go && \
go run main.go && \
git status --short -- main.go go.mod go.sum"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\"))\n if err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n  panic(err)\n }",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.26s88.56s+250.6%———
Process Start → First Event0.51s0.22s————
First Event → First Tool Call6.56s20.48s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles330%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)330%———
Output Tokens838677-19.2%———
Reasoning / Thinking Tokens388329-15.2%———
Total Input Tokens105260179977+71.0%———
Cached Input Tokens67840145920+115.1%———
Uncached Input Tokens3742034057-9.0%———
Cost0.12580.1342+6.6%———
Cached vs Uncached Token Ratio1.81:14.28:1+136.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "cat main.go; rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'go.mod' -g '"'!go.sum'"'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
cat main.go'
✓ ✓
/bin/zsh -lc 'cat main.go; go run main.go'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-09-composite-refactor

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.45s111.74s+279.4%29.37s57.39s+95.4%
Process Start → First Event0.16s0.16s—0.14s0.14s—
First Event → First Tool Call5.19s15.74s—4.80s6.29s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles58+60.0%58+60.0%
Initial Load / Discovery Turns54-20.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)58+60.0%58+60.0%
Output Tokens830980+18.1%763617-19.1%
Reasoning / Thinking Tokens218381+74.8%186129-30.6%
Total Input Tokens156106199569+27.8%116075247913+113.6%
Cached Input Tokens143360183040+27.7%104960226816+116.1%
Uncached Input Tokens1274616529+29.7%1111521097+89.8%
Cost0.08080.1041+28.8%0.06590.1188+80.3%
Cached vs Uncached Token Ratio11.25:111.07:1-1.5%9.44:110.75:1+13.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'Makefile'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'docs/**'"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat config/config.go && \
rg \"\\.Port\\b|ListenPort|Config\\{\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat config/config.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
rg --files'
✓ ✓
/bin/zsh -lc "rg -n 'Config\\{|\\.Port|DefaultConfig|ListenPort|Host:' . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
4✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
5✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git rev-parse --show-toplevel && \
git status --short -- config/config.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
8—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short && \
cat config/config.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'go.mod' -g 'go.sum' -g 'Makefile' | sort"
? ✗
/bin/zsh -lc 'cat PROJECT_BRIEF.md && \
cat config/config.go'
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\.Port|ListenPort|DefaultConfig|Config\\{' --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'config.go' -g 'go.mod'"
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
rg --files | sort | head -80'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
p = Path('config/config.go')
s = p.read_text()
s = s.replace('\\t\"archive/tar\"\\n', '')
s = s.replace('\\tPort int\\n', '\\tListenPort int\\n')
s = s.replace('func (c *Config) Addr() string {\\n\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.Port)\\n}', 'func DefaultConfig() *Config {\\n\\treturn &Config{\\n\\t\\tHost:       \"localhost\",\\n\\t\\tListenPort: 8080,\\n\\t}\\n}\\n\\nfunc (c *Config) Addr() string {\\n\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)\\n}')
tmp = p.with_suffix(p.suffix + '.tmp')
with tmp.open('w') as f:
    f.write(s)
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, p)
PY
gofmt -w config/config.go
git diff -- config/config.go"
✓ ✓
/bin/zsh -lc 'cat config/config.go'
5✓ ✓
/bin/zsh -lc 'cat config/config.go; git status --short; git diff --stat'
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost: \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
8—✓ ✓
/bin/zsh -lc 'go test ./...'

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency21.88s44.44s+103.1%23.66s81.58s+244.9%
Process Start → First Event0.52s0.18s—0.15s0.14s—
First Event → First Tool Call6.79s7.19s—4.97s5.40s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles38+166.7%511+120.0%
Initial Load / Discovery Turns34+33.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)38+166.7%511+120.0%
Output Tokens490584+19.2%630904+43.5%
Reasoning / Thinking Tokens1050-100.0%83228+174.7%
Total Input Tokens96189239414+148.9%115894193448+66.9%
Cached Input Tokens85760216576+152.5%104960168192+60.2%
Uncached Input Tokens1042922838+119.0%1093425256+131.0%
Cost0.05490.1185+115.7%0.06250.1193+91.0%
Cached vs Uncached Token Ratio8.22:19.48:1+15.3%9.60:16.66:1-30.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port|Port:' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'Makefile' | head -80"
2✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|ListenPort|config.Config|Host:|Port' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat config/config.go'
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
✓ ✓
/bin/zsh -lc "rg -n '\\bPort\\b|DefaultConfig|Config\\{' --glob '"'!AGENTS.override.md'"' ."
4—✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
5—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
7—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
8—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'config/config.go' -g 'go.mod' | sort"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\bPort\\b|DefaultConfig|ListenPort' --glob '"'!AGENTS.override.md'"'"
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null || \
true; rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' | sort"
3✓ ✓
/bin/zsh -lc "sed -n '1,100p' config/config_test.go && \
rg --files | head -60"
semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('config/config.go')
s = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY

git diff -- config/config.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc "sed -n '1,100p' config/config.go && \
git status --short"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
6—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port"
}
7—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
8—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
9—semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
10—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
11—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.09s41.46s+65.2%21.17s73.85s+248.9%
Process Start → First Event0.16s0.15s—0.27s0.38s—
First Event → First Tool Call4.44s6.78s—7.29s6.35s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles67+16.7%311+266.7%
Initial Load / Discovery Turns62-66.7%220%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)67+16.7%311+266.7%
Output Tokens732480-34.4%658909+38.1%
Reasoning / Thinking Tokens1200-100.0%43173+302.3%
Total Input Tokens175390221315+26.2%77099228685+196.6%
Cached Input Tokens158464202496+27.8%67584207616+207.2%
Uncached Input Tokens1692618819+11.2%951521069+121.4%
Cost0.09260.1037+12.0%0.04940.1181+138.8%
Cached vs Uncached Token Ratio9.36:110.76:1+14.9%7.10:19.85:1+38.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\bPort\\b|\\.Port\\b|Config\\s*\\{' --glob '"'!AGENTS.override.md'"'"
✓ ✓
/bin/zsh -lc 'cat config/config.go'
3✓ ✓
/bin/zsh -lc "rg --files | head -80 && \
rg -n 'ListenPort|DefaultConfig|default.*(host|port)|localhost|8080'"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Config.Port"
}
4✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
semedit/semantic_rename (✓ ✓)
{
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
6✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short -- config/config.go go.mod go.sum'
semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
7—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git diff -- config/config.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README.md' | head -80 && \
rg -n '\\bPort\\b|ListenPort|DefaultConfig' --glob '"'!go.sum'"' --glob '"'!go.mod'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'config/config.go' -g 'go.mod' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc 'cat config/config.go && \
cat config/config_test.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
if [ -f PROJECT_BRIEF.md ]; then cat PROJECT_BRIEF.md; fi'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, name = tempfile.mkstemp(dir=p.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
cat config/config.go
git diff -- config/config.go"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
4—semedit/semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Addr"
}
6—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
9—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
10—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
11—? ✗
/bin/zsh -lc 'git ls-files -- config/config.go; git check-ignore -v config/config.go || \
true; cat config/config.go; git diff --no-index /dev/null config/config.go'

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency16.80s39.73s+136.4%67.08s188.06s+180.4%
Process Start → First Event0.15s0.14s—0.18s0.14s—
First Event → First Tool Call4.36s6.37s—10.28s7.43s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%15+400.0%
Internal Tool Cycles46+50.0%620+233.3%
Initial Load / Discovery Turns40-100.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)46+50.0%620+233.3%
Output Tokens510472-7.5%99210356+944.0%
Reasoning / Thinking Tokens990-100.0%2464241+1624.0%
Total Input Tokens96143170598+77.4%1373832339355+1602.8%
Cached Input Tokens85760152832+78.2%1241602194944+1667.8%
Uncached Input Tokens1038317766+71.1%13223144411+992.1%
Cost0.05500.0885+60.9%0.07961.0922+1272.6%
Cached vs Uncached Token Ratio8.26:18.60:1+4.2%9.39:115.20:1+61.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSFAIL—
Oracle L3: Clean BuildPASSPASS—PASSFAIL—
Oracle L4: Verification TestPASSPASS—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n \"Config\\.Port|\\.Port\\b|type Config|DefaultConfig|port:\" --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n \"ListenPort|Host|localhost|8080|DefaultConfig\" --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
4✓ ✓
/bin/zsh -lc 'tmp=$(mktemp config/config.go.XXXXXX) && \
cat > "$tmp" <<'"'EOF'
package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
EOF
mv \""'$tmp" config/config.go
cat config/config.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config.go"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -100 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'config/config.go' -g 'go.mod' -g 'go.sum' -g 'Makefile'"
2? ✗
/bin/zsh -lc "cat config/config.go && \
rg -n '\\b\\.Port\\b|\\bPort\\s*:' --glob '"'!config/config_test.go'"' . && \
rg -n 'Config|default' PROJECT_BRIEF.md 2>/dev/null"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n 'ListenPort|DefaultConfig|Port|Host' --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'rg --files'
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time
p = Path('config/config.go')
s = p.read_text()
s = s.replace('\\t\"archive/tar\"\\n', '')
s = s.replace('\\tPort int\\n', '\\tListenPort int\\n')
s = s.replace('c.Host, c.Port)', 'c.Host, c.ListenPort)')
s = s.replace('\\nfunc (c *Config) Addr()', '\\nfunc DefaultConfig() *Config {\\n\\treturn &Config{\\n\\t\\tHost:       \"localhost\",\\n\\t\\tListenPort: 8080,\\n\\t}\\n}\\n\\nfunc (c *Config) Addr()')
fd, name = tempfile.mkstemp(prefix='.config.go.', dir=str(p.parent))
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
    os.utime(p, None)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
cat config/config.go
git status --short"
semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename may modify references in config/config_test.go, directly conflicting with the user's explicit prohibition on editing tests. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, cl
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short -- config/config.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config_test.go",
  "language": "go",
  "symbol": "TestDefaultConfig"
}
6✓ ✓
/bin/zsh -lc 'git ls-files --error-unmatch config/config.go; git check-ignore -v config/config.go || \
true; git diff --no-index /dev/null config/config.go | head -40; git rev-parse --show-toplevel'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go config/config_test.go go.mod go.sum'
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
9—✓ ✓
/bin/zsh -lc "git status --short && \
git rev-parse --show-toplevel && \
rg -n 'archive/tar|fmt|Port|ListenPort|DefaultConfig' config/config.go"
10—? ✗
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go config/config_test.go go.mod go.sum && \
git show HEAD:config/config.go'
11—semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename would modify config/config_test.go, violating the user's explicit prohibition on editing tests. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why it was blocked by
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
12—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
14—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum && \
git diff -- go.mod go.sum'
15—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
16—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
17—semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename is expected to modify the protected test file, directly violating the user's explicit instruction not to edit tests. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify w
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
18—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
19—? ✗
/bin/zsh -lc 'go test ./...'
20—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum'

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency50.48s87.53s+73.4%25.58s50.70s+98.2%
Process Start → First Event0.14s0.14s—0.29s0.17s—
First Event → First Tool Call5.91s14.35s—6.22s5.12s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles58+60.0%39+200.0%
Initial Load / Discovery Turns52-60.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)58+60.0%39+200.0%
Output Tokens901827-8.2%559819+46.5%
Reasoning / Thinking Tokens151226+49.7%138260+88.4%
Total Input Tokens135963217175+59.7%96345241668+150.8%
Cached Input Tokens124160199168+60.4%85760223488+160.6%
Uncached Input Tokens1180318007+52.6%1058518180+71.8%
Cost0.07370.1080+46.5%0.05660.1148+102.8%
Cached vs Uncached Token Ratio10.52:111.06:1+5.1%8.10:112.29:1+51.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port\\b|Config\\s*\\{' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'Makefile' -g 'docs/**'"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g '*.go' -g 'README*' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n \"Config\\.Port|\\.Port|DefaultConfig|ListenPort\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!*.txtar'"'"
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
4✓ ✓
/bin/zsh -lc 'git diff -- config/config.go go.mod go.sum && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
5? ✗
/bin/zsh -lc 'cat config/config.go; git diff --no-index /dev/null config/config.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
7—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
8—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short -- config/config.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'go.sum' && \
rg -n 'Config\\.Port|\\.Port\\b|ListenPort|DefaultConfig' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' | sort"
2✓ ✓
/bin/zsh -lc 'cat config/config.go && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat config/config.go'
3? ✗
/bin/zsh -lc "cat config/config_test.go && \
cat AGENTS.override.md && \
rg -n 'config\\.Config|Config\\{' --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
4—✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|ListenPort|Host:|Port:' . -g '*.go' -g '"'!**/*_test.go'"'"
5—✓ ✓
/bin/zsh -lc 'rg --files | head -60'
6—✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
9—✓ ✓
/bin/zsh -lc 'cat config/config.go'

Test case: task-11-mixed-sink-api-migration

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency185.84s207.10s+11.4%185.02s152.25s-17.7%
Process Start → First Event0.14s0.43s—0.25s0.13s—
First Event → First Tool Call5.38s6.61s—4.96s4.64s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%52-60.0%
Internal Tool Cycles1230+150.0%1722+29.4%
Initial Load / Discovery Turns34+33.3%53-40.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1230+150.0%1722+29.4%
Output Tokens1204211688-2.9%181914626-74.6%
Reasoning / Thinking Tokens37534531+20.7%75572136-71.7%
Total Input Tokens16135843034497+88.1%2438127816507-66.5%
Cached Input Tokens15068162817536+87.0%2290688729856-68.1%
Uncached Input Tokens106768216961+103.2%14743986651-41.2%
Cost0.84111.4495+72.3%1.26310.4836-61.7%
Cached vs Uncached Token Ratio14.11:112.99:1-8.0%15.54:18.42:1-45.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -100"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go && \
cat go.mod'
? ✗
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat README.md 2>/dev/null'
4✓ ✓
/bin/zsh -lc 'cat audit/*_test.go service/*_test.go legacy/export.go metrics/writer.go'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,200p' audit/event.go && \
sed -n '1,200p' audit/sink.go"
5✓ ✓
/bin/zsh -lc 'cat audit/summary.go cmd/migrate/main.go'
✓ ✓
/bin/zsh -lc 'git status --short'
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event_test.go && \
cat service/dispatch_test.go'
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- && \
cat audit/normalize.go audit/sink.go service/dispatch.go'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "service"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./... && \
git status --short && \
git diff -- audit/summary.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
10✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff -- service/dispatch.go audit/normalize.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "auto_organize_imports": false,
  "file": "service/dispatch.go",
  "source": "func deliver(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
11✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- go.mod go.sum service/dispatch.go audit/normalize.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- service/dispatch.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
13—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
14—✓ ✓
/bin/zsh -lc 'git diff -- service/dispatch.go audit/normalize.go && \
git status --short && \
shasum -a 256 go.mod'
15—? ✗
/bin/zsh -lc 'make check'
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git ls-files -v audit/normalize.go service/dispatch.go && \
git diff --stat && \
git status --short'
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
19—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
cat go.mod && \
git status --short'
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'git diff -- audit/summary.go; shasum -a 256 go.mod; git status --short'
23—✓ ✓
/bin/zsh -lc 'go test ./...'
24—✓ ✓
/bin/zsh -lc 'go test ./...'
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "deliver"
}
26—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
29—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "Record"
}
30—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README*' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
2✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'Makefile' -g 'go.mod' | xargs -r -n1 echo"
✓ ✓
/bin/zsh -lc 'rg --files | sort | head -200 && \
git status --short && \
cat go.mod && \
rg -n "probe|Probe|audit|Deliver|Persist|event" audit service db'
3✓ ✓
/bin/zsh -lc "cat AGENTS.override.md; rg -n \"audit|probe|EventKind|event kind|Kind\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go && \
cat AGENTS.override.md'
4✓ ✓
/bin/zsh -lc "for f in audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go metrics/writer.go legacy/export.go; do echo '---' "'$f; cat "$f"; done; ls -la'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Write"
}
5✓ ✓
/bin/zsh -lc 'rg -n "delivery|probe|normaliz|Sink|persist" docs audit service'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "journal/file.go",
  "symbol": "JournalSink.Write"
}
6✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md; cat audit/summary.go; cat go.mod; test -f Makefile && \
cat Makefile || \
true'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./...'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/transaction.go",
  "symbol": "BufferedSink.Commit"
}
8✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; git rev-parse --show-toplevel; git status --short --untracked-files=all | head -50; git ls-files audit/normalize.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
10✓ ✓
/bin/zsh -lc 'pwd; cat docs/audit-sinks.md; cat audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go; cat service/dispatch.go service/replay.go; git status --short -- audit service metrics legacy go.mod go.sum; git diff -- audit service metrics legacy go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
11✓ ✓
/bin/zsh -lc "rg -n '\\.Write\\(|Write\\(event|BufferedSink|FanoutSink|Normalize\\(' --glob '*.go' --glob '"'!**/*_test.go'"' .; find . -maxdepth 2 -type f | sort"
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go journal/file.go && \
git diff -- audit journal metrics legacy go.mod go.sum && \
go test ./...'
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
13✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go audit/event.go; git status --short -- audit metrics legacy go.mod go.sum; rg -n 'Classify|probe|health' audit --glob '*.go'"
✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
git diff -- audit/normalize.go audit/summary.go journal/file.go go.mod go.sum'
14✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./audit ./service && \
git diff --check && \
git status --short -- go.mod go.sum'
? ✗
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files audit/normalize.go audit/summary.go journal/file.go && \
git status --short --untracked-files=all | head -30 && \
git diff --no-index /dev/null audit/normalize.go'
15✓ ✓
/bin/zsh -lc "cat service/dispatch.go service/replay.go audit/normalize.go; git status --short -- service audit go.mod go.sum; rg -n '\\.Write\\(|Deliver\\(' service"
✓ ✓
/bin/zsh -lc 'git status --short -- audit go.mod go.sum journal && \
cat audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go journal/file.go'
16✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md audit/transaction.go service/dispatch.go service/replay.go audit/event_test.go audit/summary_test.go service/dispatch_test.go'
17✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
20—✓ ✓
/bin/zsh -lc 'go test ./...'
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
22—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum && \
git diff -- service/dispatch.go service/replay.go audit/normalize.go audit/summary.go journal/file.go && \
git diff -- go.mod go.sum'

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency62.65s92.74s+48.0%224.77s——
Process Start → First Event0.15s0.15s—0.25s——
First Event → First Tool Call5.11s4.64s—5.96s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%5——
Internal Tool Cycles812+50.0%14——
Initial Load / Discovery Turns21-50.0%4——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)812+50.0%14——
Output Tokens14841466-1.2%25405——
Reasoning / Thinking Tokens531549+3.4%11398——
Total Input Tokens209265394382+88.5%1948730——
Cached Input Tokens180992365056+101.7%1818112——
Uncached Input Tokens2827329326+3.7%130618——
Cost0.14110.1898+34.5%1.2411——
Cached vs Uncached Token Ratio6.40:112.45:1+94.5%13.92:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestPASSPASS—FAIL——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat go.mod && \
rg -n "event|probe|persist|delivery|Sink|Kind" service audit metrics legacy cmd'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat service/dispatch.go && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat audit/summary.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/event.go audit/sink.go audit/summary.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary_test.go'
4✓ ✓
/bin/zsh -lc "cat audit/summary_test.go service/dispatch_test.go; rg -n \"control|probe|Normalize|DispatchAll\" . -g '"'!AGENTS.override.md'"'"
✓ ✓
/bin/zsh -lc 'cat metrics/writer.go legacy/export.go cmd/migrate/main.go'
5? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile' && \
git diff --check && \
git diff -- audit/normalize.go service/dispatch.go audit/summary.go"
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- go.mod go.sum audit service metrics legacy'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go; git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9—? ✗
/bin/zsh -lc "cat audit/normalize.go && \
rg --files -g 'Makefile' -g 'makefile' -g 'Taskfile*' -g 'README*'"
10—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
11—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff --'
12—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff -- audit/normalize.go service/dispatch.go audit/summary.go && \
git check-ignore -v audit/normalize.go service/dispatch.go audit/summary.go || \
true'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
not published
2✓ ✓
/bin/zsh -lc "git status --short && \
cat docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,200p' audit/transaction.go"
not published
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' service/replay.go && \
sed -n '1,240p' db/db.go && \
sed -n '1,240p' health/health.go && \
rg -n \"Kind|probe|Write\\(\" --glob '"'!**/*_test.go'"' ."
not published
4✓ ✓
/bin/zsh -lc "sed -n '1,260p' service/dispatch_test.go && \
sed -n '1,260p' audit/event_test.go && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,240p' audit/summary_test.go && \
sed -n '1,180p' journal/file.go && \
sed -n '1,160p' metrics/writer.go && \
sed -n '1,160p' legacy/export.go"
not published
5? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile' && \
git diff -- audit service && \
git status --short"
not published
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit service'
not published
7✓ ✓
/bin/zsh -lc "git diff --stat && \
git diff --cached --stat && \
sed -n '1,100p' audit/normalize.go && \
sed -n '1,100p' service/dispatch.go && \
git status --short --untracked-files=no"
not published
8✓ ✓
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go && \
git rev-parse --show-toplevel && \
git diff HEAD -- audit/normalize.go'
not published
9✓ ✓
/bin/zsh -lc "pwd && \
git status --short && \
sed -n '1,180p' audit/normalize.go && \
sed -n '1,150p' audit/sink.go && \
sed -n '1,120p' audit/event.go && \
sed -n '1,120p' audit/summary.go && \
sed -n '1,120p' service/dispatch.go && \
sed -n '1,120p' service/replay.go && \
sed -n '1,120p' audit/fanout.go && \
sed -n '1,100p' audit/transaction.go && \
cat go.mod"
not published
10✓ ✓
/bin/zsh -lc "gofmt -w audit/normalize.go audit/fanout.go && \
go test ./... && \
git diff --check && \
git status --short && \
sed -n '1,100p' audit/normalize.go && \
sed -n '1,100p' audit/fanout.go"
not published
11✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' audit/normalize.go && \
rg -n 'probe|Classify\\(' --glob '"'!**/*_test.go'"' . && \
git status --short"
not published
12✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check'
not published
13✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go && \
sed -n '1,100p' audit/normalize.go && \
git status --short"
not published
14✓ ✓
/bin/zsh -lc "gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
sed -n '1,45p' service/dispatch.go && \
sed -n '1,35p' audit/summary.go && \
sed -n '1,35p' audit/normalize.go && \
git status --short"
not published

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency190.80s217.75s+14.1%290.02s273.49s-5.7%
Process Start → First Event0.57s0.15s—0.21s0.21s—
First Event → First Tool Call5.82s6.30s—5.49s5.75s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1319+46.2%1736+111.8%
Initial Load / Discovery Turns12+100.0%220%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1319+46.2%1736+111.8%
Output Tokens1410213007-7.8%1929624914+29.1%
Reasoning / Thinking Tokens41744669+11.9%758612817+69.0%
Total Input Tokens18416422480734+34.7%20356924762473+133.9%
Cached Input Tokens17185282316032+34.8%19043844458496+134.1%
Uncached Input Tokens123114164702+33.8%131308303977+131.5%
Cost0.96591.2117+25.5%1.14042.3462+105.7%
Cached vs Uncached Token Ratio13.96:114.06:1+0.7%14.50:114.67:1+1.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -120 && \
git status --short"
2? ✗
/bin/zsh -lc 'cat PROJECT_BRIEF.md; cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go; cat Makefile'
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc "cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go; cat go.mod; rg -n 'probe|Normalize|Record|DispatchAll' ."
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go'
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go && \
cat audit/event_test.go && \
cat audit/summary.go && \
cat audit/summary_test.go'
5✓ ✓
/bin/zsh -lc 'go test ./...; git status --short; git diff --check; git diff -- audit/normalize.go service/dispatch.go; git ls-files audit/normalize.go service/dispatch.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func Normalize(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
6✓ ✓
/bin/zsh -lc 'pwd; git status --short; git diff -- audit/normalize.go service/dispatch.go go.mod go.sum; cat audit/normalize.go service/dispatch.go'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "service/dispatch.go",
  "placement": "file_end",
  "source": "func deliver(sink audit.Sink, event audit.Event) error {\n\tevent = audit.Normalize(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
7✓ ✓
/bin/zsh -lc "cat AGENTS.override.md; cat audit/event.go audit/sink.go audit/summary.go; rg -n 'NormalizeKind|audit\\.Sink|MemorySink|probe' ."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go && \
go test ./... && \
git diff --check && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go; git diff -- audit/summary.go audit/normalize.go audit/sink.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./... && \
git diff --check && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git status --short --untracked-files=all && \
git diff --stat && \
git diff --name-only'
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go; git diff -- service/dispatch.go audit/normalize.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat service/dispatch.go && \
git diff --exit-code -- go.mod go.sum && \
git diff --exit-code -- '*_test.go' && \
go test ./... && \
git diff --check"
12✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = strings.ToLower(strings.TrimSpace(kind))\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
13✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go; go test ./...'
✓ ✓
/bin/zsh -lc "go test ./... && \
git diff --check && \
git diff --exit-code -- go.mod go.sum && \
git diff --exit-code -- '*_test.go'"
14—✓ ✓
/bin/zsh -lc "cat service/dispatch.go && \
cat audit/normalize.go && \
go test ./... && \
git diff --check && \
git diff --exit-code -- go.mod go.sum && \
git diff --exit-code -- '*_test.go'"
15—semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "Normalize"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
19—✓ ✓
/bin/zsh -lc "go test ./... && \
git diff --check && \
git diff --exit-code -- go.mod go.sum && \
git diff --exit-code -- '*_test.go'"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
2✓ ✓
/bin/zsh -lc "rg --files -g '"'!*_test.go'"' -g 'AGENTS*' -g 'README*' -g 'PROJECT_BRIEF.md' | sort && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,200p' audit/event.go"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
find . -maxdepth 2 -name 'PROJECT_BRIEF.md' -o -name 'README.md' | sort && \
cat go.mod && \
git status --short"
3? ✗
/bin/zsh -lc "sed -n '1,220p' audit/sink.go && \
sed -n '1,220p' audit/transaction.go && \
sed -n '1,240p' audit/summary.go && \
sed -n '1,200p' legacy/export.go && \
sed -n '1,180p' metrics/writer.go && \
sed -n '1,220p' service/dispatch_test.go && \
sed -n '1,220p' audit/event_test.go && \
cat Makefile"
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service"
}
4✓ ✓
/bin/zsh -lc "rg -n 'Write\\(|probe|Classify|Normalize|DispatchAll|BufferedSink|FanoutSink' --glob '*.go' --glob '"'!**/*_test.go'"' && \
cat AGENTS.override.md && \
git status --short"
✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat audit/fanout.go && \
cat audit/normalize.go'
5✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat audit/summary_test.go && \
cat audit/transaction_test.go 2>/dev/null; cat go.mod && \
find . -maxdepth 2 -type f | sort'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat service/replay.go && \
cat api/router.go'
✓ ✓
/bin/zsh -lc "cat audit/transaction.go audit/summary.go service/replay.go db/db.go metrics/writer.go legacy/export.go api/router.go && \
rg -n 'Write\\(|Kind|probe|Probe|Sink' --glob '*.go' --glob '"'!**/*_test.go'"'"
7✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
root = Path('.')
updates = {
'audit/normalize.go': '''// Package audit owns event normalization at the delivery boundary.
package audit

import (
\\t\"strings\"
)

func Normalize(event Event) Event {
\\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))
\\treturn event
}
''',
'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch Normalize(Event{Kind: kind}).Kind {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
''',
}
for name, content in updates.items():
    path = root / name
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise
PY"
✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat audit/event_test.go audit/summary_test.go service/dispatch_test.go'
8✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
replacements = {
'service/dispatch.go': [('return d.sink.Write(event)', 'return d.sink.Write(audit.Normalize(event))'), ('if err := writeEvent(sink, event); err "'!= nil {'"', 'if err := writeEvent(sink, audit.Normalize(event)); err "'!= nil {'"')],
'service/replay.go': [('if err := writeEvent(sink, event); err "'!= nil {'"', 'if err := writeEvent(sink, audit.Normalize(event)); err "'!= nil {'"')],
'audit/fanout.go': [('func (f FanoutSink) Write(event Event) error {', 'func (f FanoutSink) Write(event Event) error {\\n\\tevent = Normalize(event)')],
'audit/transaction.go': [('for _, event := range b.queue {\\n\\t\\tif err := b.Sink.Write(event); err "'!= nil {'"', 'for _, event := range b.queue {\\n\\t\\tif err := b.Sink.Write(Normalize(event)); err "'!= nil {'"')],
'audit/sink.go': [('func (m *MemorySink) Write(event Event) error {\\n\\tm.Events = append(m.Events, event)', 'func (m *MemorySink) Write(event Event) error {\\n\\tm.Events = append(m.Events, Normalize(event))')],
'journal/file.go': [('func (j *JournalSink) Write(event audit.Event) error {\\n\\tj.Entries = append(j.Entries, event)', 'func (j *JournalSink) Write(event audit.Event) error {\\n\\tj.Entries = append(j.Entries, audit.Normalize(event))')],
}
for name, pairs in replacements.items():
    path = Path(name)
    original = path.read_text()
    updated = original
    for old, new in pairs:
        if old not in updated:
            raise SystemExit(f'missing expected text in {name}: {old"'!r}'"')
        updated = updated.replace(old, new, 1)
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(updated)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise
PY
gofmt -w audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go audit/sink.go journal/file.go service/dispatch.go service/replay.go"
semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}\n\nfunc Deliver(sink Sink, event Event) error {\n\tevent.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
9✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go audit/sink.go journal/file.go service/dispatch.go service/replay.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'ls docs 2>/dev/null && \
cat docs/SUBOPTIMAL_TOOLS.md 2>/dev/null || \
true'
10✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,80p' service/dispatch.go && \
git rev-parse --show-toplevel && \
git ls-files -v audit/normalize.go && \
git diff --stat && \
git status --short --untracked-files=all"
? ✗
/bin/zsh -lc "cat docs/audit-sinks.md && \
rg --files -g 'SUBOPTIMAL_TOOLS.md'"
11✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
12✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat audit/sink.go && \
cat journal/file.go && \
cat go.mod && \
test ! -e go.sum || \
cat go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func Deliver(sink Sink, event Event) error {\n\tevent.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
13✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go audit/sink.go journal/file.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
14✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' audit/normalize.go && \
go test ./audit && \
git diff --check"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
15✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' audit/normalize.go && \
cat go.mod && \
if [ -e go.sum ]; then printf 'go.sum exists\\n'; else printf 'go.sum absent\\n'; fi && \
go test ./service && \
git diff --check"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
16✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
changes = {
'audit/normalize.go': '''// Package audit owns event normalization at the delivery boundary.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}

func Normalize(event Event) Event {
\\tevent.Kind = NormalizeKind(event.Kind)
\\treturn event
}
''',
'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch NormalizeKind(kind) {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
''',
'service/dispatch.go': None,
}
for name, content in changes.items():
    path = Path(name)
    if name == 'service/dispatch.go':
        content = path.read_text()
        old = '''func (d *Dispatcher) Record(event audit.Event) error {
\\treturn d.sink.Write(audit.Normalize(event))
}'''
        new = '''func (d *Dispatcher) Record(event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\treturn d.sink.Write(event)
}'''
        if old not in content:
            raise SystemExit('expected Dispatcher.Record body not found')
        content = content.replace(old, new, 1)
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tif err := Deliver(b.Sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
17✓ ✓
/bin/zsh -lc "git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- go.mod go.sum '*_test.go' && \
cat go.mod && \
test "'! -e go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—✓ ✓
/bin/zsh -lc "python3 -c 'import os, tempfile; p=\"docs/SUBOPTIMAL_TOOLS.md\"; d=os.path.dirname(p); fd,t=tempfile.mkstemp(dir=d, prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(\"# Suboptimal tool behavior\\n\\n- Tool: "'`semantic_insert_function`'"\\n  Target: "'`audit/normalize.go`'"\\n  Failure: rejected a snippet containing two function declarations with the diagnostic "'`multiple declarations found in snippet: expected single function declaration, found 2 declarations`.'"\\n  Workaround: inserted each function separately with the same semantic tool.\\n  Root cause: the tool accepts exactly one function declaration per call.\\n\"); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(t,p); fd=os.open(d,os.O_RDONLY); os.fsync(fd); os.close(fd)' && \
git status --short && \
cat audit/normalize.go && \
cat audit/summary.go"
19—? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile' && \
git diff --check && \
git diff -- go.mod go.sum"
20—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
21—✓ ✓
/bin/zsh -lc 'git diff --check; go test ./...; git status --short; git diff --stat; git diff -- go.mod go.sum; git diff -- audit/normalize.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go'
22—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff -- audit/normalize.go && \
git ls-files audit/normalize.go && \
git check-ignore -v docs/SUBOPTIMAL_TOOLS.md'
23—✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go audit/transaction.go; cat go.mod; if test -f go.sum; then cat go.sum; else echo 'go.sum absent'; fi; git status --short"
24—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
25—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
26—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check'
27—semedit/semantic_lookup (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
29—✓ ✓
/bin/zsh -lc 'cat go.mod; test ! -e go.sum && \
echo '"'go.sum absent'; go test ./..."
30—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
31—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
32—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
33—✓ ✓
/bin/zsh -lc 'cat go.mod; test ! -e go.sum && \
echo '"'go.sum absent'; go test ./..."
34—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
35—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
36—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
cat go.mod && \
test ! -e go.sum && \
echo '"'go.sum absent'"

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency167.53s116.77s-30.3%138.86s147.27s+6.1%
Process Start → First Event1.18s0.53s—0.16s0.16s—
First Event → First Tool Call7.07s6.08s—5.93s5.23s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns51-80.0%51-80.0%
Internal Tool Cycles1419+35.7%1332+146.2%
Initial Load / Discovery Turns43-25.0%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1419+35.7%1332+146.2%
Output Tokens110082551-76.8%142093607-74.6%
Reasoning / Thinking Tokens27751243-55.2%60571947-67.9%
Total Input Tokens1905449538326-71.7%1615671508671-68.5%
Cached Input Tokens1631232500736-69.3%1442816435968-69.8%
Uncached Input Tokens27421737590-86.3%17285572703-57.9%
Cost1.26560.2666-78.9%1.04620.3602-65.6%
Cached vs Uncached Token Ratio5.95:113.32:1+123.9%8.35:16.00:1-28.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -100"
2? ✗
/bin/zsh -lc 'cat AGENTS.override.md && \
cat PROJECT_BRIEF.md && \
rg -n "event|probe|persist|Deliver|sink" audit service metrics legacy'
✓ ✓
/bin/zsh -lc 'rg --files | sort'
3? ✗
/bin/zsh -lc "find . -maxdepth 2 -type f -not -path './.git/*' -print && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go && \
cat Makefile"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
git status --short && \
cat AGENTS.override.md'
4✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event_test.go && \
cat service/dispatch_test.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
cat go.mod'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "."
}
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./...'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
6✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
sed -n '1,160p' audit/normalize.go && \
sed -n '1,160p' audit/sink.go && \
sed -n '1,160p' service/dispatch.go && \
find . -maxdepth 1 -type f -print"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
7✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/summary.go && \
rg -n \"Normalize|IsProbe|New.*Sink|\\.Write\\(\" . -g '"'!**/*_test.go'"' && \
git diff --stat && \
git status --short --untracked-files=all"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/sink.go service/dispatch.go && \
go test ./... && \
git diff -- audit/sink.go service/dispatch.go audit/normalize.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat go.mod'
9✓ ✓
/bin/zsh -lc "git status --short --untracked-files=no && \
sed -n '1,160p' audit/summary.go && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' audit/sink.go && \
cat docs/audit-sinks.md"
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary_test.go'
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "// NormalizeKind canonicalizes event labels before delivery and classification.\nfunc NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
11✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
sed -n '1,140p' audit/normalize.go && \
sed -n '1,140p' audit/sink.go && \
git status --short --untracked-files=no"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.Classify(event.Kind) == \"control\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./...'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
13✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' service/dispatch.go && \
git status --short --untracked-files=no"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
14✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
15—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short && \
git diff --exit-code -- go.mod go.sum'
18—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff --stat && \
git status --short'
19—? ✗
/bin/zsh -lc 'git ls-files audit/normalize.go service/dispatch.go audit/summary.go; git show HEAD:audit/normalize.go; git show HEAD:service/dispatch.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*vendor*'"' | head -100 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'go.mod' -g 'go.sum' -g '*.go' | head -160"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/normalize.go && \
sed -n '1,260p' audit/fanout.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,240p' audit/transaction.go && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,220p' docs/audit-sinks.md"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'docs/**' -g '.codex/**' -g '.agents/**'"
3✓ ✓
/bin/zsh -lc "sed -n '1,260p' db/db.go && \
sed -n '1,220p' journal/file.go && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,220p' legacy/export.go && \
sed -n '1,220p' metrics/writer.go && \
rg -n \"\\.Write\\(|Record\\(|DispatchAll|BufferedSink|FanoutSink|TransactionalSink|Kind|probe\" --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
rg --files | sort'
4? ✗
/bin/zsh -lc "sed -n '1,240p' audit/event_test.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,300p' service/dispatch_test.go && \
cat go.mod && \
rg -n \"make check|verify\" . --glob '"'!**/*_test.go'"'"
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
5✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile' -g 'AGENTS*' -g 'README*' && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short"
semedit/semantic_lookup (✓ ✗): resolve Deliver: symbol not found
{
  "language": "go",
  "symbol": "Deliver"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff --stat && \
git status --short'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Write"
}
7✓ ✓
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true && \
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Commit"
}
8✓ ✓
/bin/zsh -lc "pwd && \
cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short && \
rg -n \"delivery boundary|Normalize|probe|Kind\" --glob '"'!**/*_test.go'"'"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Classify"
}
9✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum && \
git status --short'
semedit/semantic_lookup (✓ ✗): resolve Normalize: symbol not found
{
  "language": "go",
  "symbol": "Normalize"
}
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
git diff -- go.mod go.sum && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "language": "go",
  "symbol": "MemorySink.Write"
}
11✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/fanout.go",
  "language": "go",
  "symbol": "FanoutSink.Write"
}
12✓ ✓
/bin/zsh -lc "sed -n '1,120p' service/dispatch.go && \
git diff --exit-code -- go.mod go.sum && \
go test ./service"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/transaction.go",
  "language": "go",
  "symbol": "BufferedSink.Begin"
}
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('service/dispatch.go')
source = path.read_text()
source = source.replace('''\\tevent = audit.NormalizeEvent(event)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\treturn d.sink.Write(event)
''', '''\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\treturn d.sink.Write(event)
''')
source = source.replace('''\\t\\tevent = audit.NormalizeEvent(event)
\\t\\tif event.Kind == \"probe\" {
\\t\\t\\tcontinue
\\t\\t}
\\t\\tif err := sink.Write(event); err "'!= nil {
'"''', '''\\t\\tevent = audit.NormalizeEvent(event)
\\t\\tif err := sink.Write(event); err "'!= nil {
'"''')
fd, temporary = tempfile.mkstemp(prefix='.dispatch.go.', dir=str(path.parent))
try:
    with os.fdopen(fd, 'w') as output:
        output.write(source)
        output.flush()
        os.fsync(output.fileno())
    os.replace(temporary, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
finally:
    if os.path.exists(temporary):
        os.unlink(temporary)
PY
gofmt -w service/dispatch.go
sed -n '1,100p' service/dispatch.go
cat audit/normalize.go
cat audit/summary.go
git diff --exit-code -- go.mod go.sum
go test ./..."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/transaction.go",
  "language": "go",
  "symbol": "BufferedSink.Commit"
}
14—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "journal/file.go",
  "language": "go",
  "symbol": "JournalSink.Write"
}
15—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service"
}
16—✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go'
17—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "Dispatcher.Record"
}
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "DispatchAll"
}
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/replay.go",
  "language": "go",
  "symbol": "Replay"
}
20—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "language": "go",
  "symbol": "Write"
}
22—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
23—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
25—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
27—✓ ✓
/bin/zsh -lc 'go test ./...'
28—✓ ✓
/bin/zsh -lc "git status --short && \
git diff --stat && \
git diff -- go.mod go.sum '*_test.go'"
29—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go'
30—✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git ls-files audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum'
31—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/normalize.go"
}
32—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "language": "go",
  "symbol": "NormalizeKind"
}

default prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency173.04s281.33s+62.6%143.02s200.21s+40.0%
Process Start → First Event0.45s0.92s—0.18s0.15s—
First Event → First Tool Call7.67s6.79s—4.36s6.24s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1840+122.2%1526+73.3%
Initial Load / Discovery Turns35+66.7%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1840+122.2%1526+73.3%
Output Tokens1659015449-6.9%1511117443+15.4%
Reasoning / Thinking Tokens70386849-2.7%59767937+32.8%
Total Input Tokens17594463317282+88.5%21029812525155+20.1%
Cached Input Tokens15772163144704+99.4%19717122369792+20.2%
Uncached Input Tokens182230172578-5.3%131269155363+18.4%
Cost1.14521.4963+30.7%1.08471.2981+19.7%
Cached vs Uncached Token Ratio8.66:118.22:1+110.5%15.02:115.25:1+1.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—FAILFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' -g '*.md' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g 'README*' -g '*.go' -g 'Makefile' -g 'go.mod' | head -120"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' service/dispatch_test.go && \
sed -n '1,220p' audit/event_test.go && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,180p' metrics/writer.go && \
sed -n '1,180p' legacy/export.go && \
rg -n 'probe|DispatchAll|Normalize|Kind' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go'
4? ✗
/bin/zsh -lc "cat go.mod && \
rg --files -g 'Makefile' -g 'makefile' -g 'go.work' -g '.golangci*' -g 'README*' && \
"
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go legacy/export.go metrics/writer.go'
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time
root = Path.cwd()
updates = {
    'audit/normalize.go': '''// Package audit owns event-kind normalization used at the delivery boundary.\\npackage audit\\n\\nimport (\\n\\t\"strings\"\\n)\\n\\nfunc NormalizeKind(kind string) string {\\n\\treturn strings.ToLower(strings.TrimSpace(kind))\\n}\\n\\nfunc IsProbe(kind string) bool {\\n\\treturn NormalizeKind(kind) == \"probe\"\\n}\\n''',
    'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.\\npackage audit\\n\\nfunc Classify(kind string) string {\\n\\tswitch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":\\n\\t\\treturn \"control\"\\n\\tdefault:\\n\\t\\treturn \"data\"\\n\\t}\\n}\\n''',
    'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.\\npackage service\\n\\nimport \"example.com/auditapp/audit\"\\n\\ntype Dispatcher struct {\\n\\tsink audit.Sink\\n}\\n\\nfunc NewDispatcher(sink audit.Sink) *Dispatcher {\\n\\treturn &Dispatcher{sink: sink}\\n}\\n\\nfunc (d *Dispatcher) Record(event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\tif audit.IsProbe(event.Kind) {\\n\\t\\treturn nil\\n\\t}\\n\\treturn d.sink.Write(event)\\n}\\n\\nfunc DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\twriteEvent := audit.Sink.Write\\n\\tfor _, event := range events {\\n\\t\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\t\\tif audit.IsProbe(event.Kind) {\\n\\t\\t\\tcontinue\\n\\t\\t}\\n\\t\\tif err := writeEvent(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n'''
}
for name, content in updates.items():
    path = root / name
    fd, tmp = tempfile.mkstemp(prefix=path.name + '.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
        now = time.time_ns()
        os.utime(path, ns=(now, now))
    except BaseException:
        try: os.unlink(tmp)
        except FileNotFoundError: pass
        raise
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
git diff -- audit/normalize.go audit/summary.go service/dispatch.go"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Record"
}
6✓ ✓
/bin/zsh -lc "git status --short && \
git diff --stat && \
cat audit/normalize.go && \
sed -n '1,100p' service/dispatch.go && \
"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "DispatchAll"
}
7? ✗
/bin/zsh -lc 'make check'
semedit/semantic_lookup (✓ ✗): resolve NormalizeKind: symbol not found
{
  "symbol": "NormalizeKind"
}
8✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- go.mod go.sum && \
git status --short && \
git diff --no-index /dev/null audit/normalize.go || \
true'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
9✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
10✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- && \
rg -n 'Normalize|Sink|Record|Dispatch|Classify|probe|health' . -g '*.go' -g '*.md' && \
cat audit/sink.go && \
cat audit/normalize.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- go.mod go.sum && \
git status --short -- audit service metrics legacy go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
13✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
cat audit/summary_test.go && \
git diff -- audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum metrics legacy && \
git status --short && \
git diff -- audit service'
14✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff --check && \
git status --short -- go.mod go.sum audit'
? ✗
/bin/zsh -lc 'git diff --stat && \
git diff --no-index /dev/null audit/normalize.go'
15✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- service audit go.mod go.sum"
✓ ✓
/bin/zsh -lc 'pwd && \
rg --files && \
git status --short'
16✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff --check && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go docs/audit-sinks.md'
17✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/summary.go && \
sed -n '1,80p' service/dispatch.go && \
git diff -- go.mod go.sum"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "NormalizeKind"
}
18✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Record"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if audit.NormalizeKind(event.Kind) == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
21—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif audit.NormalizeKind(event.Kind) == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
22—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
23—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum && \
git status --short'
24—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum'
25—semedit/semantic_lookup (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'
28—semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Verification on path "." may format and modify all Go files, including protected tests, conflicting with the user's explicit prohibition on test edits; use a read-only or narrowly scoped verification instead. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for
{
  "path": "."
}
29—semedit/semantic_verify (✓ ✓)
{
  "path": "audit/summary.go"
}
30—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
31—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
32—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
33—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
34—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
35—semedit/semantic_verify (✓ ✓)
{
  "path": "service/dispatch.go"
}
36—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'
37—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
38—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
39—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
40—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -160"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg -n "probe|Probe|EventKind|Kind|deliver|persist|Fanout|Sink|Normalize" audit service db'
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
sed -n '1,200p' audit/sink.go && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,180p' audit/event.go && \
sed -n '1,180p' audit/fanout.go && \
rg -n 'probe|Probe|Kind|kind' --glob '*.go'"
3✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go audit/event.go audit/summary.go'
✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
sed -n '1,180p' audit/transaction.go && \
sed -n '1,160p' audit/summary.go && \
sed -n '1,120p' metrics/writer.go && \
sed -n '1,120p' legacy/export.go && \
ls -a"
4✓ ✓
/bin/zsh -lc 'cat metrics/writer.go legacy/export.go db/db.go api/router.go health/health.go'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5✓ ✓
/bin/zsh -lc "rg -n \"func .*Write|\\.Write\\(event|Sink\\.Write\" --glob '*.go' ."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
6✓ ✓
/bin/zsh -lc 'cat journal/file.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
7? ✗
/bin/zsh -lc "gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go && \
rg --files -g 'Makefile' -g 'makefile' -g 'Taskfile*'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tevent.Kind = NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := b.Sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
git status --short && \
git diff --stat && \
go test ./...'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return d.sink.Write(audit.Event{ID: event.ID, Kind: audit.NormalizeKind(event.Kind)})",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
9✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all | head -80 && \
git ls-files audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum; cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go; cat service/dispatch.go service/replay.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum && \
git diff -- audit/normalize.go service/dispatch.go service/replay.go && \
git status --short -- audit/normalize.go service/dispatch.go service/replay.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum'
? ✗
/bin/zsh -lc "git diff -- audit service && \
rg --files -g 'Makefile' -g 'makefile' -g 'go.work' && \
sed -n '1,80p' audit/normalize.go && \
git status --short"
13✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
cat audit/summary.go && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "git diff -- audit service; rg --files -g 'Makefile' -g 'makefile' -g 'go.work'; sed -n '1,80p' audit/normalize.go; git status --short"
14✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go && \
git diff -- go.mod go.sum'
? ✗
/bin/zsh -lc "sed -n '1,100p' audit/sink.go; sed -n '1,100p' audit/fanout.go; sed -n '1,100p' service/dispatch.go; git rev-parse --show-toplevel; ls -la .git"
15✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum && \
cat audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go; sed -n '1,120p' audit/sink.go; sed -n '1,100p' audit/fanout.go; sed -n '1,100p' audit/transaction.go; sed -n '1,100p' service/dispatch.go; git status --short -- audit service go.mod go.sum"
18—✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/summary.go; rg -n 'NormalizeKind|Classify\\(' --glob '*.go' --glob '"'!**/*_test.go'"'; git status --short"
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "(*Dispatcher).Record"
}
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/dispatch.go"
}
23—✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go"
24—✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/summary.go"
25—✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/normalize.go"
26—✓ ✓
/bin/zsh -lc 'go test ./...'

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency173.19s246.36s+42.2%169.00s153.19s-9.4%
Process Start → First Event2.40s0.19s—0.12s0.49s—
First Event → First Tool Call12.92s16.79s—5.67s8.08s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns51-80.0%51-80.0%
Internal Tool Cycles1419+35.7%1327+107.7%
Initial Load / Discovery Turns110%31-66.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1419+35.7%1327+107.7%
Output Tokens152822924-80.9%217533786-82.6%
Reasoning / Thinking Tokens56091582-71.8%121852164-82.2%
Total Input Tokens2453830535471-78.2%1994531460273-76.9%
Cached Input Tokens2304768502272-78.2%1811200424448-76.6%
Uncached Input Tokens14906233199-77.7%18333135825-80.5%
Cost1.21000.2649-78.1%1.33540.2700-79.8%
Cached vs Uncached Token Ratio15.46:115.13:1-2.2%9.88:111.85:1+19.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.sum'"' -g '"'!go.mod'"' | head -120"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!**/*_test.go'"' -g '"'!*test*'"' | head -80 && \
git status --short"
2✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go'
3✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go cmd/migrate/main.go'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
4? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'go.mod' -g '*.yml' -g '*.yaml' -g 'README*' && \
cat Makefile 2>/dev/null && \
git diff -- service/dispatch.go && \
git status --short"
✓ ✓
/bin/zsh -lc 'cat audit/summary.go metrics/writer.go legacy/export.go cmd/migrate/main.go'
5✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
GOTOOLCHAIN=local go test -mod=readonly ./... && \
git status --short && \
git diff --check && \
git diff -- service/dispatch.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Record"
}
6✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- service/dispatch.go audit/normalize.go audit/sink.go && \
cat audit/normalize.go audit/sink.go service/dispatch.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "DispatchAll"
}
7✓ ✓
/bin/zsh -lc 'ls -la .scratch 2>/dev/null || \
true && \
cat go.mod && \
git diff -- go.mod go.sum'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Classify"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
GOTOOLCHAIN=local go test -mod=readonly ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go && \
git diff HEAD --stat && \
git status --porcelain=v1 --untracked-files=all | head -30 && \
git ls-files audit/normalize.go service/dispatch.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git status --short -- audit/summary.go audit/normalize.go service/dispatch.go go.mod go.sum && \
git diff -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
GOTOOLCHAIN=local go test -mod=readonly ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
12✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/sink.go && \
git diff -- go.mod go.sum && \
git status --short -- service/dispatch.go audit/normalize.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
13✓ ✓
/bin/zsh -lc 'GOTOOLCHAIN=local go test -mod=readonly ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
14✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
GOTOOLCHAIN=local go test ./... && \
git diff --check && \
git diff -- go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
15—? ✗
/bin/zsh -lc "git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum && \
rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile'"
16—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
17—✓ ✓
/bin/zsh -lc 'git status --short && \
cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff --stat && \
git diff -- go.mod go.sum; true'
18—? ✗
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true && \
git show HEAD:audit/normalize.go && \
git show HEAD:service/dispatch.go'
19—✓ ✓
/bin/zsh -lc 'go test -mod=readonly ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'README.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'docs/**' && \
git status --short && \
rg -n \"audit|probe|persist|Sink|kind\" audit service db"
? ✗
/bin/zsh -lc 'cat tools/benchmark-harness/README.md && \
git status --short && \
rg -n "probe|audit|event kind|Kind|persist|Deliver|deliver|Write" audit service'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go && \
cat audit/event_test.go audit/summary_test.go audit/dispatch_test.go 2>/dev/null || \
true'
✓ ✓
/bin/zsh -lc 'rg --files | sort | head -100; git status --short'
4✓ ✓
/bin/zsh -lc "cat audit/sink.go audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/metrics.go metrics/writer.go legacy/export.go && \
rg -n '\\.Write\\(|Write\\(' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md; cat audit/event.go; cat audit/sink.go; cat audit/fanout.go; cat audit/normalize.go'
5✓ ✓
/bin/zsh -lc 'cat journal/file.go api/router.go db/db.go cmd/replay/main.go cmd/migrate/main.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/transaction.go; cat service/dispatch.go; cat service/replay.go; cat metrics/writer.go; cat legacy/export.go; cat db/db.go; cat journal/file.go'
7✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go; cat audit/summary_test.go; cat service/dispatch_test.go; cat api/router.go'
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
9✓ ✓
/bin/zsh -lc "pwd; git status --short -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum; git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go; cat docs/audit-sinks.md; cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go; rg -n \"Normalize|control|probe|Write\\(\" --glob '*.go'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\nswitch kind {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
10✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
11✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/event.go; cat audit/summary_test.go; git diff -- audit/summary.go audit/normalize.go go.mod go.sum; git status --short -- audit/summary.go audit/normalize.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\nif err := f.First.Write(event); err != nil {\n\treturn err\n}\nreturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
12✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go audit/summary.go; git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\tevent.Kind = NormalizeKind(event.Kind)\n\tif err := b.Sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
13✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
17—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/normalize.go"
}
18—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/sink.go"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/fanout.go"
}
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/transaction.go"
}
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "journal/file.go"
}
23—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/dispatch.go"
}
24—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/replay.go"
}
25—✓ ✓
/bin/zsh -lc 'go test -mod=readonly ./...'
26—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
git diff -- audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum'
27—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; git ls-files audit/normalize.go audit/summary.go; git diff --name-only HEAD; git status --short --untracked-files=all; git check-ignore -v audit/normalize.go'

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency141.48s238.63s+68.7%186.21s252.04s+35.4%
Process Start → First Event0.19s0.18s—0.12s0.18s—
First Event → First Tool Call9.47s5.72s—6.73s17.19s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1429+107.1%1836+100.0%
Initial Load / Discovery Turns330%43-25.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1429+107.1%1836+100.0%
Output Tokens1679119043+13.4%1966620110+2.3%
Reasoning / Thinking Tokens556010067+81.1%866310838+25.1%
Total Input Tokens30185063278606+8.6%27250784101619+50.5%
Cached Input Tokens28439043073536+8.1%25625603840512+49.9%
Uncached Input Tokens174602205070+17.4%162518261107+60.7%
Cost1.42691.6449+15.3%1.40101.9997+42.7%
Cached vs Uncached Token Ratio16.29:114.99:1-8.0%15.77:114.71:1-6.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSFAIL—FAILFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'AGENTS.md' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'Makefile' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -80"
2✓ ✓
/bin/zsh -lc "rg --files && \
rg -n \"audit|probe|Persist|Deliver|Normalize|kind|Kind\" . -g '*.go' -g '*.md'"
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md audit/normalize.go audit/event.go audit/sink.go audit/summary.go service/dispatch.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go'
4✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/summary_test.go audit/event_test.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
5? ✗
/bin/zsh -lc "sed -n '1,40p' audit/summary.go && \
cat go.mod && \
rg --files -g 'Makefile' -g 'makefile'"
semedit/semantic_outline (✓ ✓)
{
  "path": "service"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go cmd/migrate/main.go'
7✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
8✓ ✓
/bin/zsh -lc 'git ls-files audit/normalize.go audit/summary.go service/dispatch.go && \
cat audit/normalize.go && \
git diff --stat && \
git rev-parse --show-toplevel'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "service/dispatch.go",
  "placement": "file_end",
  "source": "func deliver(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
9? ✗
/bin/zsh -lc 'git check-ignore -v audit/normalize.go service/dispatch.go; git status --short --untracked-files=all | head -30; git diff --no-index /dev/null audit/normalize.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short --ignored=no -- go.mod go.sum audit service'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
sha256sum go.mod && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "cat audit/normalize.go service/dispatch.go && \
git status --short && \
git diff -- go.mod go.sum '*_test.go'"
12✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
go test ./audit ./service && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --stat && \
git status --short'
13✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
go test ./service ./audit && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel && \
git ls-files -- audit/normalize.go service/dispatch.go && \
git check-ignore -v audit/normalize.go service/dispatch.go || \
true && \
stat -f '%Sm %N' audit/normalize.go service/dispatch.go"
14✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go && \
git status --short -- go.mod go.sum && \
git status --short -- audit service metrics legacy'
15—✓ ✓
/bin/zsh -lc "go test ./... && \
git status --short -- go.mod go.sum && \
git status --short -- '*.go'"
16—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Classify"
}
17—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum && \
cat audit/summary.go'
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short -- go.mod go.sum'
20—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum'
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "deliver"
}
22—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short -- go.mod go.sum'
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
24—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
28—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "deliver"
}
29—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 18 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum' | head -150"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.md' | head -50 && \
git status --short && \
rg -n \"probe|Event|event|Persist|Deliver|Sink\" audit service"
? ✗
/bin/zsh -lc "cat PROJECT_BRIEF.md && \
cat go.mod && \
rg --files -g 'AGENTS.md' -g 'README*' -g 'docs/**' | sort"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/sink.go audit/normalize.go audit/fanout.go audit/transaction.go service/dispatch.go'
✓ ✓
/bin/zsh -lc "ls -la && \
git status --short && \
rg -n -i \"audit|probe|event|sink|delivery\" --glob '"'!**/*_test.go'"' ."
4✓ ✓
/bin/zsh -lc 'cat metrics/metrics.go metrics/writer.go legacy/export.go api/router.go'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
5✓ ✓
/bin/zsh -lc "cat service/replay.go && \
rg -n \"\\.Write\\(|queue|BufferedSink|FanoutSink|NormalizeKind|Deliver\" --glob '*.go' && \
git diff --"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat audit/sink.go && \
cat audit/summary.go'
6? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' && \
cat Makefile"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/fanout.go audit/sink.go audit/transaction.go service/dispatch.go service/replay.go && \
GOFLAGS=-mod=readonly go test ./...'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff --stat && \
git diff -- audit/normalize.go audit/fanout.go audit/sink.go audit/transaction.go service/dispatch.go service/replay.go && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/replay.go",
  "symbol": "Replay"
}
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all | head -40'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/fanout.go",
  "symbol": "Write"
}
10✓ ✓
/bin/zsh -lc 'git ls-files --stage -- audit/normalize.go service/dispatch.go; git check-ignore -v audit/normalize.go || \
true; git diff --no-index /dev/null audit/normalize.go | head -30; git status --short -- .'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "Write"
}
11✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum && \
rg --files audit service | sort && \
cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go'
semedit/semantic_inspect_symbol (✓ ✗): resolve Write in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790456596088111000/audit/transaction.go: symbol not found
{
  "file": "audit/transaction.go",
  "symbol": "Write"
}
12✓ ✓
/bin/zsh -lc "rg -n \"audit|normal|probe|Sink|deliver|persistence\" --glob '"'!**/*_test.go'"' ."
semedit/semantic_insert_type (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "type DeliverySink struct {\n\tSink\n}"
}
13✓ ✓
/bin/zsh -lc 'cat journal/file.go audit/summary.go audit/event.go docs/audit-sinks.md && \
cat audit/summary_test.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
14? ✗
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git status --short -- go.mod go.sum && \
git diff --no-index /dev/null audit/summary.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func (d DeliverySink) Write(event Event) error {\n\tevent.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.Sink.Write(event)\n}"
}
15✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
GOFLAGS=-mod=readonly go test ./audit && \
git status --short -- go.mod go.sum && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return \u0026Dispatcher{sink: audit.DeliverySink{Sink: sink}}",
  "file": "service/dispatch.go",
  "symbol": "NewDispatcher"
}
16✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go && \
git status --short -- go.mod go.sum && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "deliverySink := audit.DeliverySink{Sink: sink}\n\tfor _, event := range events {\n\t\tif err := deliverySink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
17✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/summary.go && \
git status --short -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "deliverySink := audit.DeliverySink{Sink: sink}\n\tfor _, event := range events {\n\t\tif err := deliverySink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
18✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
GOFLAGS=-mod=readonly go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
19—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
20—✓ ✓
/bin/zsh -lc 'go test ./...'
21—✓ ✓
/bin/zsh -lc "git status --short && \
git diff --stat && \
git diff -- audit/normalize.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum -- && \
git diff --name-only -- '*_test.go'"
22—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
cat service/replay.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all | head -30'
23—✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; else printf 'go.sum absent\\n'; fi"
24—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
25—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
26—✓ ✓
/bin/zsh -lc 'go test ./audit'
27—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "NewDispatcher"
}
29—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
30—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
31—✓ ✓
/bin/zsh -lc 'go test ./service ./audit'
32—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
33—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
34—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
35—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
36—✓ ✓
/bin/zsh -lc 'go test ./...'

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency145.37s185.47s+27.6%133.19s269.39s+102.3%
Process Start → First Event0.12s0.33s—0.23s0.15s—
First Event → First Tool Call16.16s13.05s—24.33s10.39s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%15+400.0%
Internal Tool Cycles1219+58.3%837+362.5%
Initial Load / Discovery Turns32-33.3%21-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1219+58.3%837+362.5%
Output Tokens1321014652+10.9%391021035+438.0%
Reasoning / Thinking Tokens71467959+11.4%219511848+439.8%
Total Input Tokens16627402940041+76.8%4069144739775+1064.8%
Cached Input Tokens14717442748928+86.8%3714564463616+1101.7%
Uncached Input Tokens190996191113+0.1%35458276159+678.8%
Cost1.09991.4477+31.6%0.25782.2173+760.0%
Cached vs Uncached Token Ratio7.71:114.38:1+86.7%10.48:116.16:1+54.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'README.md' | head -200"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files | sort'
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; rg --files -g 'AGENTS.md' -g 'README.md' -g '*.go' | sort"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'git status --short && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go'
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go go.mod'
✓ ✓
/bin/zsh -lc "cat audit/summary.go cmd/migrate/main.go audit/event_test.go audit/summary_test.go service/dispatch_test.go && \
find . -maxdepth 2 -type f -name 'README.md' -o -name 'PROJECT_BRIEF.md' -o -name 'AGENTS.md'"
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go audit/summary.go && \
git diff --check && \
git status --short --untracked-files=no'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "service/dispatch.go",
  "placement": "private_end",
  "source": "func deliver(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
7✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=no && \
git diff -- go.mod go.sum && \
cat audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go && \
cat docs/audit-sinks.md'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/summary_test.go && \
git diff -- go.mod go.sum && \
go test ./audit'
? ✗
/bin/zsh -lc 'ls -a && \
rg -n "''^check:|make check|''^test:" Makefile'
10✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
git diff -- go.mod go.sum && \
go test ./service"
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- audit/normalize.go service/dispatch.go && \
git status --short'
11✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go && \
git diff --stat && \
git status --short --untracked-files=all'
12✓ ✓
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc 'git status --short -- audit go.mod go.sum service && \
cat audit/normalize.go audit/sink.go service/dispatch.go'
13—✓ ✓
/bin/zsh -lc 'go test ./...'
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
15—✓ ✓
/bin/zsh -lc 'go test ./...'
16—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git status --short -- go.mod go.sum && \
go test ./...'
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "dispatcher := NewDispatcher(sink)\n\tfor _, event := range events {\n\t\tif err := dispatcher.Record(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
19—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -200 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -100 && \
git status --short && \
rg -n \"audit|probe|metrics|legacy\" --glob '*.go' ."
2✓ ✓
/bin/zsh -lc "ls -la && \
rg -n \"Audit|audit|Probe|probe|Sink|Deliver|Persist|event kind|Kind\" --glob '"'!**/*_test.go'"' ."
semedit/semantic_outline (✓ ✓)
{
  "path": "service"
}
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go metrics/writer.go legacy/export.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go api/router.go api/router_test.go db/db.go auth/auth.go cache/cache.go config/config.go models/models.go utils/utils.go logging/logging.go health/health.go docs/*'
semedit/semantic_outline (✓ ✓)
{
  "path": "journal"
}
5✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'go.work' -g 'go.mod' -g 'go.sum' -g 'README*' -g '*.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' && \
cat go.mod && \
git status --short -- . ':"'!go.mod'"' ':"'!go.sum'"'"
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat journal/file.go && \
cat audit/summary.go && \
cat metrics/writer.go && \
cat legacy/export.go'
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go && \
git diff --check && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
7✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go && \
git diff -- go.mod go.sum '*_test.go'"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
8✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git ls-files audit/normalize.go audit/summary.go && \
cat audit/normalize.go && \
git check-ignore -v audit/normalize.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nif err := f.First.Write(event); err != nil {\n\treturn err\n}\nreturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = audit.NormalizeEvent(event)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "return d.sink.Write(audit.NormalizeEvent(event))",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
13—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeEvent(Event{Kind: kind}).Kind\nswitch kind {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
14—? ✗
/bin/zsh -lc 'cat audit/normalize.go && \
cat Makefile && \
git diff --stat && \
git diff -- go.mod go.sum && \
git status --short'
15—✓ ✓
/bin/zsh -lc 'git diff -- audit service journal metrics legacy && \
git status --short && \
go test ./...'
16—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go service/dispatch.go service/replay.go journal/file.go && \
git ls-files audit/normalize.go audit/sink.go'
17—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat audit/sink.go && \
cat audit/fanout.go && \
cat journal/file.go && \
cat service/dispatch.go && \
cat service/replay.go && \
git status --short --untracked-files=no && \
git diff -- go.mod go.sum'
18—semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: semantic_verify on the entire workspace formats and may write all Go files, potentially modifying tests despite the user's explicit prohibition. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why
{
  "language": "go",
  "path": "."
}
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/normalize.go"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/sink.go"
}
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/fanout.go"
}
23—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "journal/file.go"
}
24—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/dispatch.go"
}
25—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/replay.go"
}
26—✓ ✓
/bin/zsh -lc 'go test ./...'
27—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
28—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
29—✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- go.mod go.sum'
30—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
31—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/dispatch.go"
}
32—✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff -- go.mod go.sum'
33—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
34—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\nreturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
35—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeKind(kind)\nswitch kind {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
36—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
37—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency146.00s259.65s+77.8%164.03s——
Process Start → First Event0.14s0.18s—0.47s——
First Event → First Tool Call12.76s8.75s—13.40s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%5——
Internal Tool Cycles1224+100.0%12——
Initial Load / Discovery Turns32-33.3%3——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)1224+100.0%12——
Output Tokens1367118220+33.3%19064——
Reasoning / Thinking Tokens694110872+56.6%10762——
Total Input Tokens22393353140965+40.3%1920542——
Cached Input Tokens20810242922752+40.4%1723904——
Uncached Input Tokens158311218213+37.8%196638——
Cost1.17371.6399+39.7%1.2954——
Cached vs Uncached Token Ratio13.15:113.39:1+1.9%8.77:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestFAILFAIL—FAIL——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'AGENTS*' | head -120 && \
git status --short && \
rg -n \"audit|probe|persist|delivery|event kind|EventKind|event_kind\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go && \
cat README.md 2>/dev/null || \
true && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
3✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go && \
cat go.mod && \
git status --short'
4✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '10,45p' service/dispatch.go && \
git diff --stat && \
git status --short --untracked-files=all"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5? ✗
/bin/zsh -lc 'git ls-files -v audit/normalize.go service/dispatch.go && \
git show HEAD:audit/normalize.go && \
git show HEAD:service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
6✓ ✓
/bin/zsh -lc 'rg --files && \
cat audit/normalize.go audit/sink.go service/dispatch.go audit/event.go && \
git diff -- go.mod go.sum && \
git status --short -- audit go.mod go.sum service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif audit.Classify(event.Kind) == \"control\" {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
cat audit/summary.go && \
cat go.mod && \
rg -n \"Normalize|probe|control|audit boundary|Sink\" . --glob '"'!**/*_test.go'"'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.Classify(event.Kind) == \"control\" {\n\t\tcontinue\n\t}\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum audit/normalize.go service/dispatch.go'
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
9✓ ✓
/bin/zsh -lc "cat audit/summary.go audit/normalize.go audit/event.go && \
cat go.mod && \
rg -n \"Classify|probe|health|NormalizeForDelivery\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "go test ./... && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum '*_test.go'"
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./... && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short'
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
go test ./... && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/summary.go audit/sink.go service/dispatch.go && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; fi'
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "path": "."
}
13—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
14—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
15—✓ ✓
/bin/zsh -lc "go test ./... && \
git status --short -- audit/sink.go audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum '*_test.go'"
16—✓ ✓
/bin/zsh -lc "cat audit/summary.go audit/normalize.go && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; else echo 'go.sum absent'; fi"
17—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
18—✓ ✓
/bin/zsh -lc 'go test ./audit ./service'
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
21—✓ ✓
/bin/zsh -lc 'go test ./service'
22—✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go service/dispatch.go go.mod && \
if [ -f go.sum ]; then cat go.sum; else echo 'go.sum absent'; fi"
23—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
24—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'go.sum' | head -80"
not published
2✓ ✓
/bin/zsh -lc "rg --files -g '"'!*.sum'"' | sort | head -180 && \
rg -n -i \"audit|probe|delivery|metrics|legacy export|sink\" . -g '"'!go.sum'"' -g '"'!go.mod'"'"
not published
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go && \
git status --short'
not published
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go service/dispatch_test.go audit/summary_test.go && \
cat metrics/writer.go legacy/export.go && \
cat config/config.go cmd/migrate/main.go cmd/replay/main.go'
not published
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./...'
not published
6✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum '*_test.go'"
not published
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go | head -100 && \
git rev-parse --show-toplevel && \
git ls-files audit/normalize.go audit/sink.go service/dispatch.go && \
git status --short --untracked-files=all | head -30'
not published
8✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go service/replay.go && \
cat go.mod && \
rg -n \"NewDeliverySink|NormalizeKind|\\.Write\\(event\\)|func .*Write\" --glob '*.go' ."
not published
9✓ ✓
/bin/zsh -lc 'go test ./...'
not published
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
go test ./audit'
not published
11✓ ✓
/bin/zsh -lc "sed -n '1,80p' service/dispatch.go && \
sed -n '1,60p' audit/sink.go && \
go test ./service"
not published
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./...'
not published

prefer_discover_semedit prompt · none MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsnone
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency150.64s——84.02s150.01s+78.5%
Process Start → First Event0.14s——0.14s0.19s—
First Event → First Tool Call7.21s——19.34s10.24s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns5——110%
Internal Tool Cycles14——616+166.7%
Initial Load / Discovery Turns1——13+200.0%
MCP Discovery / Schema Turns0——00—
Tool Invocations (per run)14——616+166.7%
Output Tokens18351——23473191+36.0%
Reasoning / Thinking Tokens9362——11301553+37.4%
Total Input Tokens2487683——271055530854+95.8%
Cached Input Tokens2309888——242432495616+104.4%
Uncached Input Tokens177795——2862335238+23.1%
Cost1.3684——0.17560.2713+54.5%
Cached vs Uncached Token Ratio12.99:1——8.47:114.06:1+66.1%
Oracle L1: Mutation PolicyPASS——PASSPASS—
Oracle L2: AST InvariantsPASS——PASSPASS—
Oracle L3: Clean BuildPASS——PASSPASS—
Oracle L4: Verification TestFAIL——PASSPASS—
Semantic Tool Invocation Verified (per run)N/A——N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 22 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum' | head -200"
not published
2✓ ✓
/bin/zsh -lc 'find . -maxdepth 2 -name AGENTS.md -o -name PROJECT_BRIEF.md -o -name README.md; cat go.mod; cat audit/event.go; cat audit/normalize.go; cat audit/sink.go; cat service/dispatch.go'
not published
3✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'README*' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md'; git status --short; cat audit/event_test.go; cat audit/summary.go; cat audit/summary_test.go; cat service/dispatch_test.go; cat metrics/writer.go; cat legacy/export.go; cat Makefile 2>/dev/null || \
true"
not published
4✓ ✓
/bin/zsh -lc 'cat cmd/migrate/main.go; cat audit/normalize.go; cat audit/event.go; cat audit/sink.go; cat service/dispatch.go; cat README.md 2>/dev/null; git diff --'
not published
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
not published
6✓ ✓
/bin/zsh -lc "rg -n '\\.Write\\(|Write\\(' --glob '*.go'; git diff --check; git diff -- audit/normalize.go audit/summary.go service/dispatch.go; git status --short; git diff -- go.mod go.sum"
not published
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; cat audit/summary.go; cat service/dispatch.go; git rev-parse --show-toplevel; git status --short --untracked-files=all'
not published
8✓ ✓
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go; git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true; git diff --no-index /dev/null audit/normalize.go | head -30'
not published
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
not published
10✓ ✓
/bin/zsh -lc "cat audit/normalize.go; cat service/dispatch.go; cat audit/summary.go; git status --short -- go.mod go.sum; git diff -- go.mod go.sum; rg -n 'Normalize|IsProbe|DispatchAll|Record\\('"
not published
11✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
not published
12✓ ✓
/bin/zsh -lc "sed -n '1,120p' service/dispatch.go; git status --short -- go.mod go.sum; git diff -- go.mod go.sum"
not published
13✓ ✓
/bin/zsh -lc "cat audit/normalize.go; cat audit/summary.go; sed -n '1,80p' service/dispatch.go; git status --short -- go.mod go.sum; git diff -- go.mod go.sum"
not published
14✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum && \
git diff -- go.mod go.sum'
not published
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS*.md' -g '"'!vendor'"' | head -80 && \
rg -n \"audit|probe|delivery|sink\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go && \
cat service/dispatch.go service/replay.go journal/file.go && \
git status --short && \
rg -n \"Normalize|probe|Write\\(\" --glob '*_test.go' ."
? ✗
/bin/zsh -lc "git status --short && \
sed -n '1,240p' tools/benchmark-harness/README.md"
3✓ ✓
/bin/zsh -lc "rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
cat metrics/writer.go legacy/export.go && \
rg -n \"NewDispatcher|DispatchAll|Replay|BufferedSink|FanoutSink|MemorySink|JournalSink|Classify\\(\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/event.go && \
sed -n '1,280p' audit/sink.go && \
sed -n '1,300p' audit/fanout.go && \
sed -n '1,240p' audit/transaction.go && \
sed -n '1,220p' audit/summary.go && \
rg -n \"probe|Probe|audit|EventKind|event.Kind|Kind\" --glob '*.go' --glob '"'!**/*_test.go'"' ."
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go api/router.go api/router_test.go'
✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
sed -n '1,180p' service/replay.go && \
sed -n '1,180p' journal/file.go && \
sed -n '1,180p' metrics/writer.go && \
sed -n '1,220p' metrics/metrics.go && \
sed -n '1,180p' legacy/export.go && \
sed -n '1,180p' api/router.go && \
sed -n '1,200p' README.md 2>/dev/null || \
true && \
rg --files"
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat go.mod && \
git status --short'
6✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go service/replay.go || \
true && \
sed -n '1,80p' audit/normalize.go && \
sed -n '1,40p' audit/summary.go && \
sed -n '10,35p' service/dispatch.go"
semedit/semantic_insert_structure (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "kind": "function",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7—semedit/semantic_insert_structure (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/sink.go",
  "kind": "type",
  "source": "type NormalizingSink struct {\n\tSink\n}"
}
8—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/sink.go",
  "source": "func (s NormalizingSink) Write(event Event) error {\n\tevent.Kind = NormalizeKind(event.Kind)\n\treturn s.Sink.Write(event)\n}"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "sink = audit.NormalizingSink{Sink: sink}\n\twriteEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return \u0026Dispatcher{sink: audit.NormalizingSink{Sink: sink}}",
  "file": "service/dispatch.go",
  "symbol": "NewDispatcher"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "sink = audit.NormalizingSink{Sink: sink}\n\twriteEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
13—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go service/replay.go && \
git status --short && \
git diff -- go.mod go.sum'
14—✓ ✓
/bin/zsh -lc 'go test ./...'
15—✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/sink.go && \
cat audit/summary.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,60p' service/replay.go && \
git status --short"
16—? ✗
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/sink.go service/dispatch.go && \
git check-ignore -v audit/normalize.go audit/sink.go service/dispatch.go && \
git diff --no-index /dev/null audit/normalize.go'